Recent Lag ... and what's being done.

General Comments, Questions about all things OmnipotentS that don't go in other topics/forums
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Recent Lag ... and what's being done.

Post by captainsnarf »

I wonder if HoC and Omni 2.0 are hosted on the same physical machine. :cry: :lol:
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Recent Lag ... and what's being done.

Post by pooty »

I've also been wondering how we'd do with dedicated machine. Maybe can see how things are going over at CEONSS? But they have less laggy maps, and vehicles than we do. They also I don't think see as high peak player counts looking here: http://stats.ceonss.net/index.php?stats=matches where as we can get 20-32 players which is often when lag gets most intense... BUT I've seen it with less on some days and other days 25 players and its fine, which makes me wonder about machine contention..sharing a machine.

Too bad we don't have access to the 1.0 machine.

I will say if NFO was so bad, they host HoC, DW and many other UT2004/other game servers: https://www.gametracker.com/search/?que ... =50#search
Just UT2004 only: https://www.gametracker.com/search/ut2k ... =50#search

So you'd think they'd have most of it figured out.

We could also use some of our donation money at NFO for dedicated machine for a couple months to try, to me the biggest thing is lack of access to really monitor the machine -- you get 5 minute average for CPU.
So there's this: https://www.nfoservers.com/order-virtua ... server.php Which is dedicated VM which is pretty cheap ~20/month
or dedicated hardware $100/month and up..depending. Not sure if we can find other dedicated hardware cheaper.... plus there's all the other usual BS around OS install, backups etc.

One thing we can try for cheap is put back ONSPlus for a week and see...since I think we saw much improvement with UTComp Off, and enh netcode off last time. I really, really don't want to turn off UTComp its good for the higher ping players, hitsounds, skins etc.

Also, Snarf, how much have we looking into UTComp/Vehicle interaction since its states it wasn't tested with VEhicles? I've been meaning to dig through the code and see if I can spot anything.
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Recent Lag ... and what's being done.

Post by captainsnarf »

Oh, it was broken with vehicles. It's definitely tested with vehicles now. We are the guinea pigs :lol:

If you run UTComp 1.7a on an ONS server, there is a memory leak with entering/exiting vehicles. This was found mostly by McLovin when he would play with bots. The bots would enter/exit vehicles continuously causing the problem to happen more quickly. But it would happen with players too. As the match went on, the more players entered and exited vehicles, the more memory was leaked. FPS would slowly drop over time and eventually the server would crash.

To fix this I reused the same object the player has when entering/exiting vehicles so there is no leak now - it's the same object getting reused instead of spawn/destroyed.

The next vehicle fix was to use simple cylinder collision for vehicles instead of mesh collision when ENC weapons shoot at them. Having duplicate mesh collision detection seems to cause a lot of load on the server. That last change (utcompomni 1.29?) mostly fixed the large CPU spikes caused by ENC.

I think most people including enyo had no problems with UTComp on, but ENC off on the server. It's basically ONSPlus then. None of the ENC actors are even created with ENC off on the server.
User avatar
McLovin
Posts: 1165
Joined: Sat Apr 03, 2021 12:54 pm
Location: Salt Lake City, Utah
Server Sponsor: Yes
Server Admin: Yes

Re: Recent Lag ... and what's being done.

Post by McLovin »

What's a cool-kid way of monitoring memory usage during runtime on Windose machines?

On our embedded systems at work we can monitor memory usage during runtime. If it continually dwindles over time, we know we have a leak occurring. Sometimes we have to really stress the system and run it hard overnight to get enough data to observe it.

Once, our customer had a system that had been running for 3 weeks straight before a problem would occur. They called me at home on a Saturday and my snarky response was, "reboot it." That was not the answer they wanted to hear and only infuriated them. My arse was on a plane the following Monday to site to fix it. They weren't going to turn it off until we diagnosed the problem and had a fix ready to go.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Recent Lag ... and what's being done.

Post by pooty »

What's a cool-kid way of monitoring memory usage during runtime on Windose machines?
Memory isn't an issue... we have plenty. NFO provides ganglia charts (http://ganglia.info/ )

CPU is the factor. UT is only single threaded... so single core performance matters only.. doesn't matter if its 10 core machine.

Our issue is cpu spikes... the reporting we get from NFO is on five minute averages (per their support). So it could be 20%,20%,100%, 20%, 20% would show up as 36% which wouldn't raise any concerns.

If we had direct windows access we could just watch taskmgr, or if it was linux there's much better tools like top that would tell us.

The other side is its just an engine/code limitation. Player count is hard coded capped at 32 and there's certainly a reason for it, add in lots of vehicles/projectiles/etc the engine has to work pretty hard. Sometimes a faster CPU can only compensate so much.

And our server (windows) is freshly rebooted every morning.

I think it might be worth $40-50 to spin up a dedicated VM (via NFO ) so we can better monitor and debug for a couple of months? Maybe over the holidays when our overworked staff might have some extra time?
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Recent Lag ... and what's being done.

Post by captainsnarf »

So, ENC off, Xmas stuff on, Dria with 30 players... how did it do?
Xexx
Posts: 102
Joined: Sat Jul 03, 2021 4:04 am

Re: Recent Lag ... and what's being done.

Post by Xexx »

The tree core isn't registering some hits for some reason.

https://www.youtube.com/watch?v=7_1HD28NEl0

3 point blank hits that weren't registered.
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Recent Lag ... and what's being done.

Post by captainsnarf »

Xexx wrote: Fri Dec 02, 2022 2:49 am The tree core isn't registering some hits for some reason.

https://www.youtube.com/watch?v=7_1HD28NEl0

3 point blank hits that weren't registered.
It might be the trees or it might be UTComp. I had a similar issue hitting the node in the cabinet. If I was in the corner and hitting it point blank it wouldn't register. I thought it was caused by enhanced net code, but it was off during that match so that's not it. I even added code so that when enhanced net code was on, if you shoot a node point blank with flak primary, it uses regular net code. Seems to still be broken though. It seems to only be an issue with flak primary specifically.

I tried repro'ing it with our utcomp and the original 1.7a version and I can't repro it. Maybe it *is* the trees?
User avatar
McLovin
Posts: 1165
Joined: Sat Apr 03, 2021 12:54 pm
Location: Salt Lake City, Utah
Server Sponsor: Yes
Server Admin: Yes

Re: Recent Lag ... and what's being done.

Post by McLovin »

pooty wrote: Thu Dec 01, 2022 5:28 pm ...
Memory isn't an issue... we have plenty.
,,,
Agreed, but my point was a memory loss even in small increments could be an indicator of a pointer off in the weeds, ie, a function not executing properly. When you have a big bucket of resources, it's hard to tell when it's leaking single raindrops.

At work, we were tempted to make our communication pipes huge but opted for smaller ones to quickly flag when something couldn't finish in the time required. They always flood when something bad is happening. With larger pipes, it took longer and masked the real problem.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Recent Lag ... and what's being done.

Post by pooty »

Maybe it *is* the trees?
I'll turn off the XMas stuff later today. I think those mutator only change the display not the actual collision hit box... but not 100% sure.
Post Reply