GTA:SA Resource Streaming - Revision history

Qaisjp: add category

2016-07-22T03:48:15Z

add category

← Older revision		Revision as of 03:48, 22 July 2016
Line 109:		Line 109:

	[[ru:GTA:SA_Resource_Streaming]]		[[ru:GTA:SA_Resource_Streaming]]
			[[Category: Development]]

TEDERIs at 11:37, 27 May 2014

2014-05-27T11:37:13Z

← Older revision		Revision as of 11:37, 27 May 2014
Line 107:		Line 107:

	Imagine that a very big TXD container is stored inside of GTA3.IMG. Such resources take a relatively long time to be buffered into game memory and an initialization attempt during Streaming loading may cause a lagspike, since much data is processed at once. If fibered loading is enabled, a coroutine (a.k.a. fiber) is created for every resource loading ''slicer'' with its own execution stack. When the loader attempts to read resources from the buffer, the fiber executive manager checks whether the fiber has taken [[MTA:Eir/functions/engineStreamingSetFiberedPerfMultiplier\|its time]] already. If so, it yields during the read process. Execution will continue next game frame.		Imagine that a very big TXD container is stored inside of GTA3.IMG. Such resources take a relatively long time to be buffered into game memory and an initialization attempt during Streaming loading may cause a lagspike, since much data is processed at once. If fibered loading is enabled, a coroutine (a.k.a. fiber) is created for every resource loading ''slicer'' with its own execution stack. When the loader attempts to read resources from the buffer, the fiber executive manager checks whether the fiber has taken [[MTA:Eir/functions/engineStreamingSetFiberedPerfMultiplier\|its time]] already. If so, it yields during the read process. Execution will continue next game frame.

			[[ru:GTA:SA_Resource_Streaming]]

The GTA: /* Fibered Loading */

2013-12-11T10:18:35Z

Fibered Loading

← Older revision		Revision as of 10:18, 11 December 2013
Line 104:		Line 104:

	===Fibered Loading===		===Fibered Loading===
	This is a technique utilized in MTA:Eir to improve the performance of low-end computers. It generally smoothens the FPS count of the game by removing lag-spikes. In native GTA:SA, resources are mostly loaded in one go. Using fibered loading on the other hand, loading of resources depends on the time the Streaming system may take per frame. The time it may take is derived from the total execution time of last frame.		This is a technique utilized in MTA:Eir to improve the performance of low-end computers. It generally smoothens the FPS count of the game by removing lag-spikes. In native GTA:SA, resources are mostly loaded in one go. Using [[MTA:Eir/functions/engineStreamingSetFiberedLoadingEnabled\|fibered loading]] on the other hand, loading of resources depends on the time the Streaming system may take per frame. The time it may take is derived from the total execution time of last frame.

	Imagine that a very big TXD container is stored inside of GTA3.IMG. Such resources take a relatively long time to be buffered into game memory and an initialization attempt during Streaming loading may cause a lagspike, since much data is processed at once. If fibered loading is enabled, a coroutine (a.k.a. fiber) is created for every resource loading ''slicer'' with its own execution stack. When the loader attempts to read resources from the buffer, the fiber executive manager checks whether the fiber has taken its time already. If so, it yields during the read process. Execution will continue next game frame.		Imagine that a very big TXD container is stored inside of GTA3.IMG. Such resources take a relatively long time to be buffered into game memory and an initialization attempt during Streaming loading may cause a lagspike, since much data is processed at once. If fibered loading is enabled, a coroutine (a.k.a. fiber) is created for every resource loading ''slicer'' with its own execution stack. When the loader attempts to read resources from the buffer, the fiber executive manager checks whether the fiber has taken [[MTA:Eir/functions/engineStreamingSetFiberedPerfMultiplier\|its time]] already. If so, it yields during the read process. Execution will continue next game frame.

The GTA: /* Ideas for Expansion */

2013-12-11T10:17:22Z

Ideas for Expansion

← Older revision		Revision as of 10:17, 11 December 2013
Line 102:		Line 102:

	Problem with such a proposal is that the RenderWare streaming functions are hooked by the network module.		Problem with such a proposal is that the RenderWare streaming functions are hooked by the network module.

			===Fibered Loading===
			This is a technique utilized in MTA:Eir to improve the performance of low-end computers. It generally smoothens the FPS count of the game by removing lag-spikes. In native GTA:SA, resources are mostly loaded in one go. Using fibered loading on the other hand, loading of resources depends on the time the Streaming system may take per frame. The time it may take is derived from the total execution time of last frame.

			Imagine that a very big TXD container is stored inside of GTA3.IMG. Such resources take a relatively long time to be buffered into game memory and an initialization attempt during Streaming loading may cause a lagspike, since much data is processed at once. If fibered loading is enabled, a coroutine (a.k.a. fiber) is created for every resource loading ''slicer'' with its own execution stack. When the loader attempts to read resources from the buffer, the fiber executive manager checks whether the fiber has taken its time already. If so, it yields during the read process. Execution will continue next game frame.

The GTA: /* Conversion into Engine Objects */

2013-12-03T23:29:18Z

Conversion into Engine Objects

← Older revision		Revision as of 23:29, 3 December 2013
Line 79:		Line 79:
	There are '''4 states''' that a '''slicer''' can be in: idle, buffering, loading and waiting. Idle slicers wait for requests that can occupy them. When slicers are buffering, they wait for their read requests to finish. Loading slicers wait until the game has loaded all game resources from them. Waiting is an exceptional status that is caused if a slicer is requested to push data for engine object conversion but the read requests have not finished yet.		There are '''4 states''' that a '''slicer''' can be in: idle, buffering, loading and waiting. Idle slicers wait for requests that can occupy them. When slicers are buffering, they wait for their read requests to finish. Loading slicers wait until the game has loaded all game resources from them. Waiting is an exceptional status that is caused if a slicer is requested to push data for engine object conversion but the read requests have not finished yet.

	Loading is the most interesting status. The slicers push the raw data into loader routines along with their model IDs. Using the '''model ID''' the engine knows what that raw data is supposed to represent. For instance, model id 1361 is a model resource that is stored in a .DFF file. The loader routines usually load data in one go. If data is found too big, it can be '''loaded across multiple slicer pulses''' (the slicer resides in '''loading status'''). Then the data is processed in '''coroutine fashion'''. Natively, only '''texture dictionaries''' that exceed a certain block count use this feature. Also, the engine has been optimized to assume that all data is loaded in two slicer pulses.		Loading is the most interesting status. The slicers push the raw data into loader routines along with their model IDs. Using the '''model ID''' the engine knows what that raw data is supposed to represent. For instance, model id 1361 is a model resource that is stored in a .DFF file. The loader routines usually load data in one go. If data is found too big, it can be '''loaded across multiple slicer pulses''' (the slicer resides in '''loading status'''). Then the data is processed in '''coroutine fashion'''. Natively, only '''texture dictionaries''' that exceed a certain block count use this feature. Also, the engine has been optimized to assume that all data is loaded in two slicer pulses (max.).

	Each slicer has its own '''syncSemaphore''' from the streaming runtime. The syncSemaphore index corresponds to the slicer index.		Each slicer has its own '''syncSemaphore''' from the streaming runtime. The syncSemaphore index corresponds to the slicer index.

The GTA: /* Ideas for Expansion */

2013-11-29T16:40:54Z

Ideas for Expansion

← Older revision		Revision as of 16:40, 29 November 2013
Line 97:		Line 97:

	==Ideas for Expansion==		==Ideas for Expansion==
	Natively, the loader does not operate with perfect efficiency. As stated above, the system mimicks coroutine behavior. If very big .DFF models are loaded, the Streaming system can cause FPS drops. The way texture containers are loaded is imperfect, even if their loading is split into two pulses. The theory the conversion of every texture into a game instance is a complex task. Natively, streaming is not dependant on timing of the load process. If it were, then the system could yield when a TXD container takes too long to load.		Natively, the loader does not operate with perfect efficiency. As stated above, the system mimicks coroutine behavior. If very big .DFF models are loaded, the Streaming system can cause FPS drops. The way texture containers are loaded is imperfect, even if their loading is split into two pulses. The theory is that the conversion of every texture into a game instance is a complex task. Natively, streaming is not dependant on timing of the load process. If it were, then the system could yield when a TXD container takes too long to load.

	My proposal is that every slicer obtains a fiber in a MTA execution manager. Once per frame (and at all other plausible moments) all fibers will be pulsed, thus progressing resource loading. Since every fiber maintains its own runtime stack, complex code is avoided. When a certain quota of data is read from a RenderWare stream, the fiber's execution is yielded. Such a change in resource loading is going to smoothen the FPS of GTA:SA when traveling across the world.		My proposal is that every slicer obtains a fiber in a MTA execution manager. Once per frame (and at all other plausible moments) all fibers will be pulsed, thus progressing resource loading. Since every fiber maintains its own runtime stack, complex code is avoided. When a certain quota of data is read from a RenderWare stream, the fiber's execution is yielded. Such a change in resource loading is going to smoothen the FPS of GTA:SA when traveling across the world.

	Problem with such a proposal is that the RenderWare streaming functions are hooked by the network module.		Problem with such a proposal is that the RenderWare streaming functions are hooked by the network module.

The GTA at 16:32, 29 November 2013

2013-11-29T16:32:09Z

← Older revision		Revision as of 16:32, 29 November 2013
Line 97:		Line 97:

	==Ideas for Expansion==		==Ideas for Expansion==
	Natively, the loader does not operate with perfect efficiency. As stated above, the system mimicks coroutine behavior. If very big .DFF models are loaded, the Streaming system can cause FPS drops. The way texture containers are loaded is imperfect, even if their loading is split into two pulses.		Natively, the loader does not operate with perfect efficiency. As stated above, the system mimicks coroutine behavior. If very big .DFF models are loaded, the Streaming system can cause FPS drops. The way texture containers are loaded is imperfect, even if their loading is split into two pulses. The theory the conversion of every texture into a game instance is a complex task. Natively, streaming is not dependant on timing of the load process. If it were, then the system could yield when a TXD container takes too long to load.

	My proposal is that every slicer obtains a fiber in a MTA execution manager. Once per frame (and at all other plausible moments) all fibers will be pulsed, thus progressing resource loading. Since every fiber maintains its own runtime stack, complex code is avoided. When a certain quota of data is read from a RenderWare stream, the fiber's execution is yielded. Such a change in resource loading is going to smoothen the FPS of GTA:SA when traveling across the world.		My proposal is that every slicer obtains a fiber in a MTA execution manager. Once per frame (and at all other plausible moments) all fibers will be pulsed, thus progressing resource loading. Since every fiber maintains its own runtime stack, complex code is avoided. When a certain quota of data is read from a RenderWare stream, the fiber's execution is yielded. Such a change in resource loading is going to smoothen the FPS of GTA:SA when traveling across the world.

	Problem with such a proposal is that the RenderWare streaming functions are hooked by the network module.		Problem with such a proposal is that the RenderWare streaming functions are hooked by the network module.

The GTA at 22:03, 28 November 2013

2013-11-28T22:03:01Z

← Older revision		Revision as of 22:03, 28 November 2013
Line 67:		Line 67:
	\|}		\|}

	[[File:Res_loader_sys.png\|~~800x640px~~]]		[[File:Res_loader_sys.png\|960px]]

	===(Threaded) Resource Reading===		===(Threaded) Resource Reading===

The GTA: added a picture for better understanding

2013-11-27T19:18:09Z

added a picture for better understanding

← Older revision		Revision as of 19:18, 27 November 2013
Line 66:		Line 66:
	\|This resource is currently being loaded in coroutine fashion. Data that is associated with it is too big to be loaded in one go.		\|This resource is currently being loaded in coroutine fashion. Data that is associated with it is too big to be loaded in one go.
	\|}		\|}

			[[File:Res_loader_sys.png\|800x640px]]

	===(Threaded) Resource Reading===		===(Threaded) Resource Reading===

The GTA at 18:06, 27 November 2013

2013-11-27T18:06:33Z

← Older revision		Revision as of 18:06, 27 November 2013
Line 45:		Line 45:

	The internal CModelInfoSA array represents each resource entry in .IMG files. The engine supports 26310 resource IDs in total.		The internal CModelInfoSA array represents each resource entry in .IMG files. The engine supports 26310 resource IDs in total.

			====Possible resource loading statuses====
			{\|class="wikitable"
			!Name
			!Description
			\|-
			\|MODEL_UNAVAILABLE
			\|The resource has not been loaded yet. No game resource is loaded for this entry. This resource is not used by any part of the game engine.
			\|-
			\|MODEL_LOADED
			\|This resource has been loaded by the Streaming system. It is ready to be used by the game engine.
			\|-
			\|MODEL_LOADING
			\|This resource resides on the to-be-loaded queue. It waits to be picked up by the Streaming system for reading from disk.
			\|-
			\|MODEL_QUEUE
			\|A slicer is currently reading data from disk for this resource.
			\|-
			\|MODEL_RELOAD
			\|This resource is currently being loaded in coroutine fashion. Data that is associated with it is too big to be loaded in one go.
			\|}

	===(Threaded) Resource Reading===		===(Threaded) Resource Reading===