Random Server Crashes..

General Comments, Questions about all things OmnipotentS that don't go in other topics/forums
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

I am going to turn off the custom scoring for now and see if that fixes the issue...if it does we know where to go fix the crash with custom scoring.

I've got some fixes I think for custom scoring and the focus/crash issue, and it might be more than just turning it off but have to test them out a bit. Ultimately, I think we can get custom scoring working right and not crash, but if we can't I think it might be easier to have 2pt games, and either accept the occasionaly 1 round 2pt match, or set the OT Time to like 2 minutes, and reduce core drain speed. So pretty much all matches are OT 1pt rounds.
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Random Server Crashes..

Post by captainsnarf »

pooty wrote: Thu Mar 23, 2023 12:01 am I am going to turn off the custom scoring for now and see if that fixes the issue...if it does we know where to go fix the crash with custom scoring.
Yes I think this is the way to go. Commenting out a couple lines here and there is too slow in isolating the problem. Whole sections need to be removed until it's found. Divide and conquer.

It could end up being something fundamental with the engine about how game rules are handled. It could be something really dumb. Can you spot the bug?

Code: Select all

function Destroyed()
{
    if(Trail != None)
        Trail.Destroy();
        
    super.Destroy();
}
A subtle bug like that could cause weird crashes too. Hard to find though.

Thank you for looking into this. I've been pretty busy lately.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

It could end up being something fundamental with the engine about how game rules are handled. It could be something really dumb. Can you spot the bug?
Yes, that's what I am thinking, there's something in Rules recursion/calling that either isn't returning properly or is not ending/destroying in the right order. Or calling Destroy on something thats already Destroyed....

I think the key will be to get the round to end "normally" to set the focus and what not without any special work and slowly add in back from there.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

So one thing I've narrowed down, working on EvenMatch custom score.
Level.GRI.Teams[Scorer.Team.TeamIndex].Score -= oldscore;
Level.GRI.Teams[Scorer.Team.TeamIndex].Score += newscore;
if(Level.GRI.Teams[Scorer.Team.TeamIndex].Score < 0)
Level.GRI.Teams[Scorer.Team.TeamIndex].Score = 0;

Screws with the timer. Meaning if you kill the core in regulation, it starts the next round timer with the remaining round time from the first round. When that runs out it doesn't go into OT. This looks like at least to me something we need to fix. That right there tells me some where it isn't getting reset.... almost like score changes, something fires elsewhere, and then we change the real score.. and something fubars.

Good news is other than that, I seem to have it working with much less code. I am testing completely without UTComp for now, locally and haven't had the need for the additional SetFocus stuff.

I am almost wondering if we just pull custom scoring out of EvenMatch. I think for now that should help with these weird crash issues.

I think we implement the custom scoring as seperate mutator, and I think we can do it easier:

On ONSOnslaughtGame:

Code: Select all

event InitGame(string Options, out string Error)
{
   
        // Register all the PowerCores
        if (N.IsA('ONSPowerCore'))
        {
            PowerCores[x] = ONSPowerCore(N);
            PowerCores[x].NotifyUpdateLinks = UpdateLinks;
            PowerCores[x].OnCoreDestroyed = MainCoreDestroyed;
MainCoreDestroyed Function is, where scoring is hard coded (gasp!).

Code: Select all

function MainCoreDestroyed(byte T)
{
    local Controller C;
    local PlayerController PC;
    local int Score;

    if (bOverTime)
        Score = 1;
    else
        Score = 2;

    if (T == 1)
    {
        BroadcastLocalizedMessage( class'ONSOnslaughtMessage', 0);
        TeamScoreEvent(0, Score, "enemy_core_destroyed");
        Teams[0].Score += Score;
        Teams[0].NetUpdateTime = Level.TimeSeconds - 1;
       CheckScore(PowerCores[FinalCore[1]].LastDamagedBy);
    }
    else
    {
        BroadcastLocalizedMessage( class'ONSOnslaughtMessage', 1);
        TeamScoreEvent(1, Score, "enemy_core_destroyed");
        Teams[1].Score += Score;
        Teams[1].NetUpdateTime = Level.TimeSeconds - 1;
        CheckScore(PowerCores[FinalCore[0]].LastDamagedBy);
    }

    //round has ended
    for (C = Level.ControllerList; C != None; C = C.NextController)
    {
        PC = PlayerController(C);
        if (PC != None)
        {
            PC.ClientSetBehindView(true);
            PC.ClientSetViewTarget(PowerCores[FinalCore[T]]);
            PC.SetViewTarget(PowerCores[FinalCore[T]]);
            if (!bGameEnded)
                PC.ClientRoundEnded();
        }
        if (!bGameEnded)
            C.RoundHasEnded();
    }

    ResetCountDown = ResetTimeDelay;
}
Think its possible to make a mutator that basically says (in PostNetBegin play?)

Code: Select all

 x = 0;
    for(N = Level.NavigationPointList; N != None; N = N.NextNavigationPoint)
    {
           // Register all the PowerCores
        if (N.IsA('ONSPowerCore'))
        {
               PowerCores[x].OnCoreDestroyed = MainCoreDestroyed_withOmniCustomScoring;
if (PowerCores[x].bFinalCore)
                FinalCore[PowerCores[x].DefenderTeamIndex] = x;
            x++;
    }
We should ideally only need to replace that function and two lines at that.

But bottom line I think I am going to rebuild EvenMatch for now without any custom scoring. That seems to be problematic for now in my local testing. And I'm not sure that EvenMatch is the right spot anyway, I think we just put it there since we're customizing it anyway, but custom scoring really isn't part of team balancing.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

I have a seperate mutator that replaces the MainCoreDestroyed:

Code: Select all

function PostBeginPlay()
{
	
	log(Class$" build "$Build, 'OmniONSScoring');
     ONSGame = ONSOnslaughtGame(Level.Game);
}	
	
function bool CheckReplacement(Actor Other, out byte bSuperRelevant)
{
	 local ONSPowerCore PC;
	  
	  PC = ONSPowerCore(Other);
	  if (PC!=None)    {
        	  PC.OnCoreDestroyed = MainCoreDestroyedOmniScoring;
            if (bDebug) log("Setting OnCoreDestoryed = Omni",'OmniONSScoring');
	}
	return true;
}

Scoring works as expected. MainCoreDestroyedOmniScoring is pretty much cut/copy from stock UT MainCoreDestroyed except the hard coded scores...

But Still have the clock/timer issue though... just like the other one. WTF?

However this is IMO, much cleaner for custom scoring. I'll play with it a bit more.

As far as the server, Custom Scoring is turned off, which should stop the crashes at the expense you can now have 2pt matches...I'll work on removing the custom scoring stuff from EvenMatch.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Cleaned up version of UTCompOmni and EvenMatchOmni .. without any Custom Scoring code are on the server.

I left times same, so you can get 2pt victory if you kill the core < 4mins. So its kind of like a mulligan but you get a new map.

I'll work on the custom scoring, but I have a feeling the custom scoring was causing crashes, because the end of round timer wasn't getting set right ever....so there's some event somewhere not getting called or as Snarf mentioned some kind of Timer (it thinks round is still going) issue.

Hopefully this stabilizes things until I fix the custom scoring....its possible it might be a ton of work, in which case I suggest we just play with round time/OT time/core drain.

And slight off topic. I see that now I can replace functions (not just values) on any class. I suppose its not surprising since under the hood its C++, but still handy thing to have in the toolbox.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

So no crashes but we got a few rounds with the funky running around and not getting the focus set. Which makes no sense, since the round stuff is now all stock... There's nothing to interfere with it..unless there's something in UTComp that messes with it unintentionally?

I mean even Team DM matches change the focus at the end and UTComp worked with those right?
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Snarf, any ideas on: few rounds with the funky running around and not getting the focus set. ? I can't duplicate it on my testing. I run dedicated server and other client. Running UTCompOmni, EvenMatchOmni on my dedicated host everything works perfectly. Now running UT2004 client (64-bit) so its a completely seperate process from the server.
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Random Server Crashes..

Post by captainsnarf »

Have you pushed the latest to github? I'll check it out when I get off work.

'MainCoreDestroyed' in ONSOnslaughtGame is what stops the running around thing.

Before it stops the running around, it does a call to CheckScore. This was where the custom scoring was hooked into. I was careful to return whatever true/false value that super.CheckScore() did, but I think that might be wrong. I think we *always* want to return false.

Another issue is inside CheckScore(), there is a GotoState('RoundEnded') or something like that. I think that transition may not be happening sometimes, or it does happen which causes it to skip the remaining code in MainCoreDestroyed method.

Either way, all of the issue is inside that MainCoreDestroyed method.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

I'll load up the CustomScoring mutator I wrote into Git.

It just replaces the MainCoreDestroyed function which is identical except for the score values, that is its a direct copy from base onslaught code (some changes for scope). It works for scoring. But when a round ends early function executes, gives 1 pt instead of two, but the base UT timer isn't reset. I suspect there's some code elsewhere that handles that reset. I am thinking of just resetting it at the end of main core destroyed in my mutator...



But my original question was even without any custom scoring, using the stock code right now on the server, without any custom scoring, we sometimes don't get end of round setfocus stuff done.. which is strange because its all now base Onslaught code. Which I can't duplicate locally.
Another issue is inside CheckScore(), there is a GotoState('RoundEnded') or something like that. I think that transition may not be happening sometimes, or it does happen which causes it to skip the remaining code in MainCoreDestroyed method.
Yes there's some timing off... somewhere. I think I could use my Mutator to log what happens where, now that we can change 'MainCoreDestroyed'
Post Reply