Random Server Crashes..

General Comments, Questions about all things OmnipotentS that don't go in other topics/forums
Post Reply
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Random Server Crashes..

Post by pooty »

So something is up. We've had way too many random server crashes..and not even on new maps, but maps that are many years old.
There's nothing to indicate any errors in the UT logs. I did open a ticket with the time/date from tonights crash to see if they have any additional info.

I suspect its something in one of the mutators, Zounds, UTComp but that's just wild ass guess.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

And the logs all abruptly end
Nothing else particular in the logs other than no UT errors..and they usually just abruptly stop in mid log
eg.

Code: Select all

EvenMatchOmni: GamePPH: PlayerName: UTComp_ONSPlayerReplicationInfo PPH: 125.80
EvenMatchOmni: GamePPH: PlayerName: UTComp_ONSPlayerReplicationInfo PPH: 166.79
EvenMatchOmni: GamePPH: PlayerName: UTComp_ONSPlayerReplicationInfo PPH: 38.16
EvenMatchOmni: GamePPH: PlayerName: UTComp_ONSPla
It doesn't even finish writing the last line..on UT errors it will not only write that but a whole error message/shutdown. These recent crashes all have that abrupt termination. That's like a hard crash vs. UT game knowing somethign went wrong.
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Random Server Crashes..

Post by captainsnarf »

The last crash tonight, I played a zound (it played) and then server was crashed. Hard to say as it could just be coincidence.

What's changed recently?
New version! Download here UTCompOmni_1.37.zip

Changes
- Add PawnCollisionHistoryLength config option. This is used to determine how much time travel history is recorded when back tracking collision detection for enhanced net code. Default is 0.35 seconds. Lower values will decrease server load at the cost of enh net code accuracy, depending on player's ping.
That was Sunday. There's no reason that would cause crashes that I can think of, but it is something that's changed recently.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

I think we start a process of elimination. It doesn't seem to be map related.

- Turn off enhanced Netcode entirely. This will allow us to check two things at once - Random Crashes, and I still see lag I swear was never there before enhanced netcode.
- If it still crashes, turn off UTCompOmni and put back ONSPlusOmni
- Turn off Zounds
Then look at other things like EvenMatch etc.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

12/9/2022 - UTComp on, EhnNetCode disabled. Will run this weekend like that and reassess here. I know some really like the ping compensation, but we are working to make sure that isn't causing crashes or lags. Maybe Sunday/Monday we will review and see what next steps are.
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Random Server Crashes..

Post by captainsnarf »

Sounds good. Take your time, no rush to turn on ENC.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

So lag was much better... but darn thing still crashed twice while I was on.
I am going to turn off UTComp tomorrow and put ONSPlusOmni back on. If we get no crashes there its something in UTComp, not related to NetCode (since it still crashed with ENC off). Perhaps something with the scoreboard/round restart, since the crash happens always at end of round--that seems to be the only consistent thing.
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Random Server Crashes..

Post by captainsnarf »

There were errors in the log for the CheckScore function (scorer is none). That is called when you score a point / kill a core. I made a change to that function not long ago to try and fix the issue where score wasn't always recorded in a fast match. I wonder if that's it. I think that was in EvenMatch though, not UTComp. I think they both have an override for CheckScore.
User avatar
EmanReleipS
Posts: 42
Joined: Wed Jun 09, 2021 1:11 am

Re: Random Server Crashes..

Post by EmanReleipS »

CEONSS is still on UTComp version 1.34 and no weird crashes, so if it is UTComp related, it is connected to something that was added/changed thereafter.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Hmm Maybe its EvenMatch then.

Code: Select all

TankMeUp-Randomizer-V15.EvenMatchRules (Function EvenMatchOmni.EvenMatchRules.CustomScore:0142) Accessed None 'Scorer'
Warning: EvenMatchRules ONS-MinusTankMeUp-Randomizer-V15.EvenMatchRules (Function EvenMatchOmni.EvenMatchRules.CustomScore:014A) Accessed None
Warning: UTComp_ONSGameRules ONS-MinusTankMeUp-Randomizer-V15.UTComp_ONSGameRules (Function UTCompOmni.UTComp_ONSGameRules.CheckScore:01C7) Accessed None 'C'
I'll take a look at it. The logs often stop in EvenMatch.. and this log from Slated shows:

Code: Select all

EvenMatchOmni: GamePPH: PlayerName: UTComp_ONSPlayerReplicationInfo Initial PPH: 291.34Multiplier 1.40 applied =407.87
EvenMatchOmni: GamePPH: PlayerName: UTComp_ONSPlayerReplicationInfo Initial PPH: 107.38Multiplier 1.00 applied =107.38
EvenMatchOmni: GamePPH: PlayerName: UTComp_ONSPlayerReplicationInfo Initial PPH: 237.88Multiplier 1.20 applied =285.46
EvenMatchOmni: GamePPH:End   Red:2563.42 Blue:2419.64
Warning: EvenMatchRules ONS-SlatedWorld-BigAl-V2.EvenMatchRules (Function EvenMatchOmni.EvenMatchRules.CustomScore:00B2) Accessed None 'Scorer'
Warning: EvenMatchRules ONS-SlatedWorld-BigAl-V2.EvenMatchRules (Function EvenMatchOmni.EvenMatchRules.CustomScore:00BA) Accessed None
Warning: EvenMatchRules ONS-SlatedWorld-BigAl-V2.EvenMatchRules (Function EvenMatchOmni.EvenMatchRules.CustomScore:00F9) Accessed None 'Scorer'
Warning: EvenMatchRules ONS-SlatedWorld-BigAl-V2.EvenMatchRules (Function EvenMatchOmni.EvenMatchRules.CustomScore:0101) Accessed None
Warning: EvenMatchRules ONS-SlatedWorld-BigAl-V2.EvenMatchRules (Function EvenMatchOmni.EvenMatchRules.CustomScore:0142) Accessed None 'Scorer'
Warning: EvenMatchRules ONS-SlatedWorld-BigAl-V2.EvenMatchRules (Function EvenMatc
Frequently it stops spitting out the PPH stuff, but that looks to be just before CustomeScore. (I'll fix the PRI Name thing too)
Post Reply