This week, we’re adding a brand new set of Brick Signs, with four variants for you to craft, adorn and place around your base. We’ve also got some insight for you into our development process after we had a perforce outage last week that impacted our development timelines. Finally, we’ve got some details on an ongoing feature development - a solo pause option. 
This Week: Brick Sign Set This week, we’re adding a new set of Brick Signs, with four different variants for you to craft. These are unlocked at Tier 3 and crafted on the Advanced Masonry Bench. Ten clay bricks, two concrete mixes, and four iron nails are required for each sign. The four variants come in floor and wall options, with tall and angled variants for use in different terrains and settings. Just like all other signs, these can be edited to display your own text or icons.
You may have noticed in last week's patch notes that we also improved how signs appear in-game by removing the artificial glow and making them look less digital. The full note is below:
This Week: Perforce
Last week two Perforce crashes happened on two days in succession. Functionally, this turned our 5 day work week into only 3 days for our normal weekly patch.Perforce is the source control system we use for Icarus, keeping the developers working in alignment on the code and assets required to make all the different versions of the game (at minimum each week is a new version). Historically, our Perforce instance has been running (not up-time) since June 2016, with our first Icarus code commit seen as 2019.
Since those first commits, the source control has grown to 6TB of project data. We use a Google Drive to keep all the full sized Icarus art assets, so the true scale of the project files is much more. The 6TB is just the raw foot print of the source code and binary assets to build the game for distribution, with the latter accounting for the majority of the 6TB. And then to ‘serve’ the data effectively to developers and build servers bring along additional requirements.The reason any of this is part of a blog post is because one of our servers experienced a “Blue Screen of Death” recently and caused corruption in our Perforce database files. Bringing Perforce back up required multiple database reads, rebuilds, and verifications. Last week two crashes happened on two days in succession, significantly effecting operations. As developers, we mitigated this by using the time to playtesting on steam and in a few cases juggled feature work in “offline” mode.
Hosting the 6TB of critical data means a lot more than a PC with a few SSDs. Huge failover batteries protect our whole office site, multiple high IO drives for RAID, spares, and backups, and we need NVMe drives to push the architectural limit to serve this data fast over the local network to multiple people. Many syncs early in the morning can saturate our 10Gb uplink.Since we’re covering our live data footprint, we should also mention that for backuping up the data we provision 20TB of HDD storage so we can have daily backups, snapshots over time, and space left over to take a full fresh backup on demand. But you can’t just have one back up, you actually need a duplicate of your backup, so there’s another 20TB. Mirror that all again, as we must save it all in another form off site in the the cloud, so we provision another 20TB of cold archival storage in a datacenter that is in geographical region outside of New Zealand. You can see how just the 6TB critical data for Icarus has quickly escalated to requiring 100TB of provisioned storage for its stable existence.Recently, rapid system halts and “blue screen of death” has caused us data corruption in our Perforce database, to the point Perforce is turned off for repair and the developers have to immediately adjust their work schedule. What does repairing Perforce involve? Perforce by design works and manages our files, the versions, and meta history for those, through several dozen paired databases. When the system halts and those databases are open or are being written, they break and they break in ways that are very bad for productivity. Luckily, there are specific Perforce tools and commands that are run as a check on the database file pairs and we keep documented in our internal wiki the process.
Fixing the corrupted perforce instance has a few steps. After the first attempted read of the database set, all the corrupt files are identified. A second pass is done to write out new replacement databases identified from the last successful checkpoints. These new replacement databases are replaced in their directory and another different integrity check is run on all the databases. This step is very slow and can take a several hours. If everything goes well, there are no more corrupt database files found. But if we find another corruption in this check, we have to redo the whole process until all corruptions are fixed.
Ultimately, the Perforce service is returned to working condition, with developers and the build servers now all syncing at the same time. Now the developers are happy again and continue to work unimpeded in Unreal and the build servers can go back to assembling builds for you.
Coming Soon: Solo Pause
We’ve been working on a new feature for Solo play. It allows players to pause their session rather than the game continuing to run in the background. This is quite a large change and requires layers of experimentation and testing internally to iron out any bugs or issues that arise. For example, multiple mission chains have to be tested as a pausing feature may impact (or impact, vice versa) their logic. At this point, we’re putting out early iterations of this change into our experimental branch, which gives us a much larger testing pool to catch bugs or oversights that slip through the cracks. We’ll be ready to release this onto the main branch once we are comfortable it is at a standard we are proud of. A quick note: this will not work on multiplayer or dedicated servers. This is due to the presence of other players and their own sessions, as well as the logic required to run a dedicated server.
Next Week: Chef Backpack
Next week, we’re adding a new ‘Chef’ Backpack for culinary arts specialists. This will allow prospectors to craft better foods with longer-lasting buffs. Alongside a decrease in the food consumption requirement, there is also a buff to water consumption, so you’ll be able to last longer without two of three pillars of your prospector's well-being.
Your support makes these updates possible.
https://store.steampowered.com/bundle/40692/Icarus_Pets_Bundle/https://store.steampowered.com/bundle/35727/Icarus_Complete_the_Set/
Changelog v2.2.51.135901
New Content [expand type=details expanded=false]
This Week: Brick Sign Set This week, we’re adding a new set of Brick Signs, with four different variants for you to craft. These are unlocked at Tier 3 and crafted on the Advanced Masonry Bench. Ten clay bricks, two concrete mixes, and four iron nails are required for each sign. The four variants come in floor and wall options, with tall and angled variants for use in different terrains and settings. Just like all other signs, these can be edited to display your own text or icons.
You may have noticed in last week's patch notes that we also improved how signs appear in-game by removing the artificial glow and making them look less digital. The full note is below:Originally posted by author: Update Sign widget material to use Decal domain which allows it to take on normals of sign mesh and handles lighting better, removing emissive glow, making the signs look less digital and cohesive within the scene
This Week: Perforce
Last week two Perforce crashes happened on two days in succession. Functionally, this turned our 5 day work week into only 3 days for our normal weekly patch.Perforce is the source control system we use for Icarus, keeping the developers working in alignment on the code and assets required to make all the different versions of the game (at minimum each week is a new version). Historically, our Perforce instance has been running (not up-time) since June 2016, with our first Icarus code commit seen as 2019.
Since those first commits, the source control has grown to 6TB of project data. We use a Google Drive to keep all the full sized Icarus art assets, so the true scale of the project files is much more. The 6TB is just the raw foot print of the source code and binary assets to build the game for distribution, with the latter accounting for the majority of the 6TB. And then to ‘serve’ the data effectively to developers and build servers bring along additional requirements.The reason any of this is part of a blog post is because one of our servers experienced a “Blue Screen of Death” recently and caused corruption in our Perforce database files. Bringing Perforce back up required multiple database reads, rebuilds, and verifications. Last week two crashes happened on two days in succession, significantly effecting operations. As developers, we mitigated this by using the time to playtesting on steam and in a few cases juggled feature work in “offline” mode.Hosting the 6TB of critical data means a lot more than a PC with a few SSDs. Huge failover batteries protect our whole office site, multiple high IO drives for RAID, spares, and backups, and we need NVMe drives to push the architectural limit to serve this data fast over the local network to multiple people. Many syncs early in the morning can saturate our 10Gb uplink.Since we’re covering our live data footprint, we should also mention that for backuping up the data we provision 20TB of HDD storage so we can have daily backups, snapshots over time, and space left over to take a full fresh backup on demand. But you can’t just have one back up, you actually need a duplicate of your backup, so there’s another 20TB. Mirror that all again, as we must save it all in another form off site in the the cloud, so we provision another 20TB of cold archival storage in a datacenter that is in geographical region outside of New Zealand. You can see how just the 6TB critical data for Icarus has quickly escalated to requiring 100TB of provisioned storage for its stable existence.Recently, rapid system halts and “blue screen of death” has caused us data corruption in our Perforce database, to the point Perforce is turned off for repair and the developers have to immediately adjust their work schedule. What does repairing Perforce involve? Perforce by design works and manages our files, the versions, and meta history for those, through several dozen paired databases. When the system halts and those databases are open or are being written, they break and they break in ways that are very bad for productivity. Luckily, there are specific Perforce tools and commands that are run as a check on the database file pairs and we keep documented in our internal wiki the process.
Fixing the corrupted perforce instance has a few steps. After the first attempted read of the database set, all the corrupt files are identified. A second pass is done to write out new replacement databases identified from the last successful checkpoints. These new replacement databases are replaced in their directory and another different integrity check is run on all the databases. This step is very slow and can take a several hours. If everything goes well, there are no more corrupt database files found. But if we find another corruption in this check, we have to redo the whole process until all corruptions are fixed.
Ultimately, the Perforce service is returned to working condition, with developers and the build servers now all syncing at the same time. Now the developers are happy again and continue to work unimpeded in Unreal and the build servers can go back to assembling builds for you.
Coming Soon: Solo Pause
We’ve been working on a new feature for Solo play. It allows players to pause their session rather than the game continuing to run in the background. This is quite a large change and requires layers of experimentation and testing internally to iron out any bugs or issues that arise. For example, multiple mission chains have to be tested as a pausing feature may impact (or impact, vice versa) their logic. At this point, we’re putting out early iterations of this change into our experimental branch, which gives us a much larger testing pool to catch bugs or oversights that slip through the cracks. We’ll be ready to release this onto the main branch once we are comfortable it is at a standard we are proud of. A quick note: this will not work on multiplayer or dedicated servers. This is due to the presence of other players and their own sessions, as well as the logic required to run a dedicated server.
Next Week: Chef Backpack
Next week, we’re adding a new ‘Chef’ Backpack for culinary arts specialists. This will allow prospectors to craft better foods with longer-lasting buffs. Alongside a decrease in the food consumption requirement, there is also a buff to water consumption, so you’ll be able to last longer without two of three pillars of your prospector's well-being.
Your support makes these updates possible.
https://store.steampowered.com/bundle/40692/Icarus_Pets_Bundle/https://store.steampowered.com/bundle/35727/Icarus_Complete_the_Set/
Changelog v2.2.51.135901
New Content [expand type=details expanded=false]
- Added sign blueprints and item setup
- Resave items datatable for sign grouping
- Update new sign item variant grouping
- Add new sign talents and recipes
- Unlocked Brick Sign recipes, item and talent for release
- Added temporary brick sign icon
- Fixed Beehive Processing VFX to replicate on client
- Add translatable text to Operations banner on C0NT4CT device
- Adding interact audio for when opening up a dead prospectors inventory. Adding to base class
- Fix spelling of Juvenile on Desert Training talent description
- Fixed Taxidermy Knife being called Trophy Knife
- Fixing typo spelling of Juvenile in talent tree
- Adding the correct ammo types to the dead bodies in the GH RG Base A Location
- Dead researchers in GH_APE_A now correctly hide the items in their hands when looted
- Fixed a number of typos in mission objectives for GH chains
- Added implementation of Swamp Ape Attractor, and localised attraction mechanic
- Added ability for generic actors to act as spawn tethers via a new interface
- Fixed radar material on swamp ape attractor not turning off when unpowered
- Adding interact sound for the great hunt device so its not just silence when you enter the UI screen
- More various updates and improvements to GH UI audio
- Adding goop dialogue line into dialogue data table. Fixes to some dialogue events where things were cut off abruptly, updating shorter mission complete line where appropriate
- Initial setup of prototype Drone AI
- Adjust name of Icy Mammoth -> Enzyme-Enhanced Mammoth in the Bestiary
- First pass on Strange Troop mission rework: Moved location to nearby cliff. Add Garganutan Jnr lair spawn where they now spawn from. Add poison cloud surrounding area to encourage antipoison pill usage and stackable poison modifier. Players must destroy lair and any spawned Jnr Apes to complete the mission
- Show instanced level exits on the compass (only as there is no map)
- Updated the icy mammoth bestiary image to match the new textures
- Cliff and Landscape Pass in Swamp, Purple/Orange Quad, Elysium
- Setting PM on wood corner stairs thats used in GH Lab base so that the footsteps played are wood and not default which is usually grass - dirt sounding
- Putting ape lab door on its hinges - on the wall so players dont open it through the ground. Visually it looks like the ape broke out through the bloody broken wall
- Elysium - cliff and ledge placement, decal and foliage painting in geothermal, red quad
- Added FTs for Stick trees (vars 1-3)
- Mammoth now sprints to arena after being hit by bullet
- Shifted the ice chunks on the back of the icy mammoth in the character and carcass BPs
- Mammoth now sprints to arena after being hit by bullet
- Move any attached arrows to corpse inventory when various Great Hunt bosses die
- Added trophies for the icy viscid and icy mammoth
- Adding updated text for the graves in the GH RG mining outpost A
- Adding legendary gauntlet aint - power up loop audio, events and notifys
- Change particle system attachment for Legendary SledgeHammer Impact Ribs
- Temporarily adjusting mission talents to reroute instead of mutually exclusive to prevent missions from being blocked. Can undo once fix is in place
- Added first pass art assets for the geothermal stick tree to be use for blocking
- Small update to the opening and closing the fort gate used in GH mission to make it feel more appropriate. Small adjustments to scorpions volumes in world to make sure they werent too intense compared with their vocalisations
- Adding various UI sounds for great hunts and a few that were missing from the main menu for no reason. Added missing hover over various items and adding select difficulty audio and biolab menu and item clicked etc
- Elysium - Added V2 blockout mesh for Iris Biolab to Eden
- Modifying Talent System to now support different talent types such as mutally exclusive selection which is required for great hunts, setting up new mutally exclusive talent connectors for all the great hunt mission trees
- Added PRP_Bench_PSU
- Fix DT Validation
- Elysium - cliff and ledge placement, decal painting in geothermal, red quad
- Removing ducking from various weapons that were causing the ambience to lower in a way that was not subtle enough. Especially noticeable during the rock golem earthquakes or by rivers etc
- Added emissive textures for the exploder bat and the ice dart bat
- GOAP Tweaks for baby apes (no longer chase birdies)
- Great Ape: More balance for large numbers of players
- Great Ape: Wire Norex world stat
- Add new world stats for rewards for great hub ape missions
- Great Ape: More ongoing balance tweaks
- Potential fix for instanced levels where everyone dies, leaves and then terminates the server instance before save happens
- Fixed bug where clicking RTCS / Quit / Unstuck wouldn't work until you unpaused the game
- Pass over all dialogue lines used in GH missions. Adjusted any dialogue that felt off, adding unique log voices, sharpened up timing etc
- Fixed issue with ESC-menu pause that prevented other players from joining while paused
- Elysium - cliff and ledge placement, decal painting in geothermal, red quad
- Adding correct death collision audio event to ape juvis so they play the death fall sound
- Disable shadow casting on invisible weather culling mesh spawned from BP_CaveInstance which was causing shadow artifacts to appear on cliff faces near cave entrances
- Initial implementation of the new Alpha Wolf art assets
- Add various new ape related world stats to mission outcomes (stats still not wired)
- Cliff and Landscape Pass in Swamp, Purple/Orange Quad, Elysium
- IM_C3 removed debug testing logic
- Geothermal material instances for various meshes
- IM_C3 mammoth can no longer be killed, is tethered to it's spawn location, fades out after reaching arena location, freezes correctly if a player causes the tile to be unloaded
- Fixing issues with legendary sledge hammer playing spear equip sound when aiming.
- Did a pass on spear equips to make sure all of them worked subtley and woudlnt cause issues under different circumstances
- Fix Ape Norex World stat isn't quite working as intented
- Added SK files, as well as Materials and Textures and BP, for Desert Bounder. Implimentation still in progress
- Great hunts selection is now data driven from new DT. Added in changing item displays
- Ensure GeoThermal pool VFX is distance culled
- Wire Ape Expuragtion world stat
- Wire Ape Escape world stat
- Fix instanced level compass exit does not show up for clients
- Wire Ape Blackmarket world stat
- Update BP_Crevasse to use Atmospheres enum to drive material overrides for future expandability
- Adding more GH UI main menu sounds. Adding the hover for upgrade slots, and select for upgrade slots
- Added WiP material override functionality to BP_crevasse
- Elysium - landscape sculpting, cliff and crevasse placement in Geothermal, red quad
- BP_Crevasse - assigned material overrides for geothermal and volcanic
- Added altenative fur setups for the alpha wolf to be compared for performance
- Pausing the audio when the game is paused for better indication of whats happened
- Update Ape_B mission to spawn a creature with the item, instead of having the item already on the ground
- Fixed stones in pickable oxite mesh not rendering properly. Fixed LODs
- Cliff and Landscape Pass in Swamp, Purple/Orange Quad, Elysium
- Adjusting where the GH device interract sound plays from to ensure it plays each time
- Further BP setup for Bounder and Carcass, as well as a second pass on the gFur. Passing on to Joe to do the final BP and Data Table setup
- Wire Ape Strange Troop world stat