51	TECH28	1.2.10	Failure of a harness or rigging element that would result in total loss of string (string is stuck during drop, or installation cannot continue and de-installation cannot happen, instrumentation already installed is abandoned)	String Total Loss	1/15/2021	1/25/2022		Low	High	Very High	Very High	Moderate	Moderate	Moderate	Installation Mgr	FS3	Mitigate	Operator Error/Equipment Failure/Hardware Failure	Set robust mechanical requirements (design and proof test) for each element (harness, pressure vessels, rigging pieces, yale grips). Contingency time accounted for each installation step in hole lifetime calculation and prescribed drill charts.	Mitigation greatly reduces probability. String could not be replaced so no cost or schedule risk, only loss of performance.	$0				0				3%	0	$ - 0	High	0	$-	Low	FS3 end
52	TECH29	1.2.10	Dust Logger winch failure	Dust Logger Winch	12/13/2018	1/25/2022		Low	Low	Low	Moderate	Low	Low	Moderate	Installation Mgr	FS3	Mitigate	Technical problems or Operator Error	Test Winch and Dust Logger Comms as soon as they are delivered to Pole, and at tower once electrical connections are established. If maulfunction happens prior to Dust Logger run start: establish Abort Time Prior to Start, postpone Dust Logger Run to next (last hole).	Last hole will need to be made larger to accomodate dust logger run. low estimate of 110 gal (difference between a deep cold ream hole and a hot + log hole) => 110*25.45= $2800. High estimate 10% fuel (800 gal) to account for possible re-ordering of holes.	$2,800	$2,800		$20,360	0	0	0		25%	0	$ 2,800.00	Low	0	$700.00	Low	FS3 end
53	TECH30	1.2.10	String Installation Winch Failure - The TU20 winch fails during installation operations. TU-20 needs to be swapped out	TU-20 Failure	12/13/2018	1/25/2022		Low	Low	Moderate	Low	Low	Moderate	Low	Installation Mgr	FS3	Mitigate	Technical problems or Operator Error	Inspect and repair both TU-20's to full operation. Create Installation Cable Swap-Out Procedure.	<1-day delay - would be absorbed in the FS	$ - 0				0	0	0		10%	0	$ - 0	Low	0	$-	Low	FS3 end
54			1.3 Deep Ice Sensor Modules
55	EXT9	1.3.1	If electronics parts for the mDOM Mainboard cannot be procured in time, the mDOM production and testing will be delayed	mDOM Electronics Supply Chain		1/27/2022		High	Moderate	Low	Low	High	Moderate	Moderate	OM Lead	Until procurement	Mitigate	External	Continuously monitor market throughout collaboration and centrally collect information of availability of critical components Place orders / publish calls for tender as early as possible Develop plans for series integration of half-mDOMs, add in mainboards as late as possible	Air shipping to New Zealand might be required if parts are sufficiently delayed. Off project cost. Critical path schedule delay would be zero if air shipping is utilized as long as parts show up by mid-2023. Cost risk is off-award	$ - 0	$0	$0	$0	26	12	52		75%	26	$ - 0	Low	19.5	$-	Low
56	EXT10	1.3.1	If suppliers don't have sufficient capacity to 3D-print mDOM PMT Support Structures, the mDOM production and testing will be delayed	mDOM Support Structure Supply Chain		1/27/2022		Moderate	Moderate	Low	Low	Moderate	Moderate	Moderate	OM Lead	Until procurement	Mitigate	External	Foresee production of PMT Support Structures in several batches that can be tendered to different manfufacturers	Air shipping to New Zealand might be required if parts are sufficiently delayed. Off project cost. Critical path schedule delay would be zero if air shipping is utilized as long as parts show up by mid-2023. Cost risk is off-award	$ - 0	$0	$0	$0	4	0	26		5%	4	$ - 0	Low	0.2	$-	Low
57	TECH33	1.3.1	If mDOM series production does not reach the projected rate of 9 mDOMs / site / week, the shipping date for the mDOMs for Strings 87+88 will be missed	mDOM Production Yield		1/27/2022		Low	Moderate	Low	Low	Moderate	Low	Low	mDOM Lead	Until OM shipping	Mitigate	Technical problems	Produce mDOM test batches at DESY and MSU as early as possible to excercise integration and FAT procedures see also risk ORG5	Air shipping to New Zealand might be required if parts are sufficiently delayed. Off project cost. Critical path schedule delay would be zero if air shipping is utilized as long as parts show up by mid-2023. Cost risk is off-award	$ - 0	$0	$0	$0	4	0	26		10%	4	$ - 0	Low	0.4	$-	Low
58	EXT11	1.3	If integrated sensors have to be stored for too long before installation, the vacuum seal might leak and re-sealing might become necessary	Sensor Long-Term Storage		6/23/2022		Low	High	Low	Low	Moderate	Low	Low	OM Lead	Until installation readiness review	Mitigate		Regularly monitor the inside pressure of a sample of stored sensors Store sensors such that they are accessible for monitoring (not McM?)	If stored sensors are accessible for regular monitoring, the problem can be detected early and sensors can be un-sealed and re-sealed at a rate of about 5 sensors / day. Off project cost	$ - 0	$0	$0	$0	8	4	12		5%	8	$ - 0	Very low	0.4	$-	Low
59	TECH34	1.3.2	If during D-Egg FAT a high failure rate or design flaw is found, reworking D-Eggs will lead to significant cost and schedule slippage	D-Egg FAT Yield		1/27/2022		Low	Low	Low	Moderate	Low	Low	Moderate	D-Egg Lead	Until FAT	Mitigate		Commence D-Egg FAT as early as possible Work closely with D-Egg team on executing and interpreting FAT	Even a minor re-work of all (300) sealed D-Eggs is a major undertaking, requiring un-sealing, re-sealing and re-testing all modules. Off project cost	$ - 0	$0	$0	$0	26	12	52		5%	26	$ - 0	Low	1.3	$-	Low
60	EXT12	1.3.5	If one of the Special Devices is not delivered, the string design must be updated and verified	Special Device Cancellation	1/15/2022	1/27/2022		Very High	Low	Low	Very low	Moderate	Moderate	Moderate	OM Lead, installation Lead	Until installation readiness review	Mitigate		Start early developing contingency plans how to replace or bypass missing Special Devices during installation	Can be managed in close to real time, perhaps a couple of days schedule slip on the ice or slight risk of degraded performance	$ - 0	$0	$0	$0	0	0	0		50%	0	$ - 0	Very low	0	$-	Low
61	TECH35	1.3.3	Because upgrade penetrator is different than Gen1 penetrator, the refurbished PDOMs must have the Gen1 penetrators removed and re-sealed with upgrade penetrators. Replacing penetrator may introduce subtle flaws that could cause PV flooding after installation at depth.	PDOM PV leak	1/15/2022	6/23/2022		Low	Moderate	Moderate	Low	Moderate	Moderate	Low	PDOM Lead	Until after freeze-in	Mitigate	No comms with deployed PDOM	Inspection of sealing surfaces, comparison of Gen1 and Gen2 PCA sealing surfaces, High Pressure testing.	If detected before installation of remaining strings, a short work stoppage for analysis of the problem might be required costed as a two-day delay. If sealing defects are sporadic, it may result in some number of PDOMs not surviving installation at depth.	$ 20,000.00	$ 10,000.00	$ 20,000.00	$ 30,000.00	0	0	1		5%	0	$ 20,000.00	Low	0	$1,000.00	Low
62	EXT18	1.3.1	If mDOM planned main board electronics parts remain unavailable, a mainboard redesign will be required: this includes hardware design effort (off Project budget) and software & firmware effort (on Project budget).	mDOM redesign	4/6/2022	6/23/2022		High	Moderate	High	Low	High	High	Moderate	mDOM lead	Calendar 2022	Mitigate	Parts delivery delays/unavailable parts	Begin planning and design work ahead of risk trigger on the new mainboard design	On project cost: need 1 FTE-year of software effort and 1/2 FTE-year of firmware work.	$ 240,000.00	$ 160,000.00		$ 320,000.00	52	26	78		85%	52	$ 240,000.00	Low	44.2	$204,000.00	Low
63	TECH36	1.3	Because NPX is very dry, sensor electronics could be damaged by ESD during xDOM testing or installation at NPX	xDOM Electronics ESD Damage	1/15/2022	6/23/2022		Low	Low	Moderate	Low	Low	Moderate	Low	OM Lead, Installation Lead	Until OM installation	Mitigate	Modules failures, procedure errors during installation	Employ appropriate internal ESD protection circuitry in the xDOM and ICM. Ensure that ESD mitigation is used on floors, people, and equipment at NPX. Ensure charge neutralization is emplloyed by procedure and equipment during NPX testing. Ensure by equipment design and operational procedure that charge neutralization is present between electrical cables, TOS and xDOMs as they are being deployed.	If process problems with ESD at South Pole, a work stoppage for renewing procedures might be required. Costed as a week delay. 1-week delay for full drill team - additional labor costs	$ 98,560.00	$ - 0	$ 98,560.00	$ 197,120.00	1	0	2		2%	1	$ 98,560.00	Low	0.02	$1,971.20	Low
64	EXT20	1.3	Because there is critical in-kind contributed labor from DESY, MSU, and Chiba to complete the mDOMs and D-Eggs, the project would be impacted if this labor were lost	Loss of In-Kind Labor (WBS 1.3)	6/27/2022										OM Lead	Until OM shipping	Accept	Loss of key personnel		Estimate 1 FTE-year of engineering labor, 3 month delay from onboarding. The main risk areas are mDOM Mainboard engineering and mDOM integration and testing coordination at DESY and MSU. Assuming a 15% probability that one of these three critical roles will have to be replaced	$ 150,000.00	$ 50,000.00		$ 250,000.00	13	4	26		15%	13	$ 150,000.00	Low	1.95	$22,500.00	Low
65	TECH53	1.3.1, 1.3.2	Because there is no long-term experience with the Shin-Etsu optical gel, non-desirable degradation of the optical gel of the mDOMs and D-Eggs might be observed only late in the project.	Optical Gel Long-Term Performance	7/1/2022	7/1/2022									OM Lead	During the full detector lifetime	Mitigate	Degradation of optical gel	Long-term testing of DVT OMs in a freezer at nominal operations temperature.	Rework of some modules before shipping might lead to delays. Cost is off-award. 1 FTE-year of a PostDoc to improve calibration if modules with different-than-expected optical performance are installed.	$ 100,000.00	$ 50,000.00		$ 200,000.00	13	0	26		5%	13	$ 100,000.00	Moderate	0.65	$5,000.00	Low
66			1.4 Comms, Power, and Timing (CPT) Distribution System
67	TECH37	1.4.1.1	Because the MCA prototype has not completed mechanical testing, we may need to revise the main cable design, which will require additional design and testing effort and delay the start of production.	Main cable design iteration	1/27/2022	3/1/2022		Low	High	Low	Very low	Moderate	Low	Low	1.4 L2 Lead	Summer 2022	Accept (near-future)	Unsatisfactory prototype test	N/A	Schedule impact depends on magnitude of changes required -- an additional prototype cycle could take 1/2 year, while minor changes could be completed in a few weeks. Cost impact limited to on-project engineering labor, likely ~1 FTE-month	$ 15,000	$0		$30,000	20	6	40		0%	20	$ 15,000.00	0	0	$-	0
68	TECH38	1.4.1	Because the MCA prototype has not completed mechanical testing, we may need to add a separate co-installed strength member to the Downhole Cable Assembly, which will increase installation complexity and require additional equipment.	Strength member needed	1/27/2022	3/1/2022		Moderate	High	Low	Very low	High	Moderate	Low	1.2 L2 Lead	Summer 2022	Mitigate	Unsatisfactory prototype test	Begin preparing design for co-deployed rope and test prototype in advance of cable tests	Estimated cost of ropes ($18k/string) is largely offset by reduced need for YaleGrips (approx. 60 omitted per string @ $270 each). Cost of additional winch etc. unknown. Time required for design of second winch system unknown. Rough guesses inserted.	$ 25,000	$10,000		$60,000	4	2	12		25%	4	$ 25,000.00	0	1	$6,250.00	0
69	TECH39	1.4.1	Because the MCA prototype has not completed mechanical testing, we may need to redesign the string hardware and installation procedures, which will require additional design effort, prototyping, and equipment testing.	Deployment procedures	1/27/2022	3/1/2022		Low	Moderate	Very Low	Very Low	Moderate	Low	Low	1.2 L2 Lead	Summer 2022	Accept (near-future)	Unsatisfactory prototype test	N/A	Design work plus prototyping/testing	$ 5,000	$2,500		$10,000	6	4	12		10%	6	$ 5,000.00	0	0.6	$500.00	0
70	TECH40	1.4.1.1	Because the MCA prototype has not completed mechanical testing, we may need to select an alternate Main Cable supplier, which will necessitate project support for cost and/or impact communications and timing performance	Alternate cable	1/27/2022	3/1/2022		Low	Very high	Very high	Moderate	Moderate	Moderate	Moderate	1.4 L2 Lead	Summer 2022	Mitigate	Unsatisfactory prototype test	Pursue development of alternate suppliers, procure cable samples where feasible	Likely to be delays in switching to new main cable supplier, most vendors are less responsive than Hexatronic.	$ 1,500,000	$300,000		$3,500,000	26	4	52		0%	26	$ 1,500,000.00	Moderate	0	$-	Low
71	TECH41	1.4.1.1	Because a MCA prototype has not yet been built, the overall diameter of the cable may be larger than estimated and not fit on the currently planned shipping/installation spool, which will require redesign of the spool, cable, and/or installation procedure and equipment	Cable too large	1/27/2022	3/1/2022		Very low	Moderate	Low	Very Low	Low	Low	Low	1.2 L2 Lead	Until first MCA is complete (Aug 2023)	Accept	Prototype cable diameter is too large. Breakouts are too large	Minimize cable diameter to preserve room on spool for breakout connections	Design work, larger spools, potentially different equipment. Cost impact is a rough guess.	$ 50,000	$10,000		$100,000	8	4	26		5%	8	$ 50,000.00	0	0.4	$2,500.00	0
72	TECH42	1.4.1.1	Because a MCA prototype has not yet been built, the overall weight of the cable may be larger than estimated and exceed the limit of the currently planned shipping pallet and/or aircraft, which will require redesign of the cable, spool, or shipping pallet.	Cable too heavy	1/27/2022	3/1/2022		Very low	Moderate	Low	Very low	Low	Low	Low	1.4 L2 Lead	Summer 2022	Accept	Prototype cable is too heavy	N/A	Possible redesign of shipping pallet, requiring additional liaison with ASC and drill lead	$ 30,000	$7,500		$45,000	16	4	24		2%	16	$ 30,000.00	0	0.32	$600.00	0
73	TECH43	1.4.1.2	Because a BCA supplier has not been identified, we may need to make significant design changes, which will increase costs and could delay production of other cable elements	BCA design iteration	1/27/2022	3/1/2022		Low	High	Moderate	Very low	Moderate	Moderate	Low	1.4 L2 Lead	2022	Mitigate	Issues during preliminary design	Engage with potential suppliers as early as possible	Restarted design phase could take several months, require several FTE-month of engineering effort. Cost of actual BCAs unlikely to be affected	$ 25,000	$5,000		$75,000	16	2	26		5%	16	$ 25,000.00	0	0.8	$1,250.00	0
74	TECH44	1.4.2	Because of connector supply chain disruptions, we may not be able to ship the SCAs or SJBs on the scheduled vessel, which will require them to be shipped ComSur the following year or eliminate the option of staging at McMurdo the season before installation.	SCA delay	1/27/2022	3/1/2022		Very low	Low	Low	Very low	Low	Low	Low	1.4 L2 Lead	Until connectors are received	Mitigate	Connector delivery delayed beyond Aug 2022	Order connectors as early as possible (order placed May 2021). Complete first cables using Gen1 leftover connectors to validate assembly procedures. Test and utilize compatible connectors from alternative suppliers	Cost of Comsur shipping estimated based on past shipments of large cables. Schedule impact central estimate reflects RDD for Comsur, low estimate is delay consistent with FY23 vessel, high estimate is shift to FY24 vessel	$ 7,500	$6,000		$15,000	36	16	64		15%	36	$ 7,500.00	0	5.4	$1,125.00	0
75	EXT13	1.4.3	Because of electronics supply chain disruptions, there may be delays in production of rev1 and/or rev2 FieldHubs, which would reduce the number of prototyping cycles available before we need to enter production.	FieldHub delay	1/27/2022	3/1/2022		Moderate	High	Very Low	very low	High	Low	Low	DESY	June 2023 (start of FieldHub production)	Mitigate	Delays in production of rev1/2 FieldHubs	Finalize selection of FPGA and other long-lead components as early as possible, order as soon as practicable. Eliminate prototyping cycles rather than delaying production (FieldHub design is already quite mature)	Elimination of prototyping cycle removes schedule impact at cost of slightly elevated technical risk	$ - 0	$0		$0	0	0	16		50%	0	$ - 0	Low	0	$-	Low
76	EXT14	1.4.4.1	Because the vendor identified for the timing and communications hardware used in the FieldHub was recently acquired, the interface to the hardware may change or the product may be retired, which will require additional hardware development effort to change to a different timing node.	White Rabbit supplier	1/27/2022	3/1/2022		Low	High	Low	very low	Moderate	Low	Low	1.4.4 L3 Lead	Spring 2023	Mitigate	Substantial changes to baseline timing solution	Identify alternate suppliers or alternate timing fanout designs that can be implemented using parts currently on hand	Alternative designs identified. Additional effort of 1±1 FTE-month required for design.	$ 15,000	$0		$30,000	4	0	8		10%	4	$ 15,000.00	Very low	0.4	$1,500.00	Low
77	ORG9	1.4.4.2	Because of increased mDOM and DEgg power consumption and remaining uncertainties around maximum device power consumption, the power distribution system may need to be redesigned for a higher DC voltage, which will require additional development effort.	DOM power limit	1/27/2022	3/1/2022		High	High	Low	Very low	High	Moderate	Low	1.4.4 L3 Lead	April 2022	Redesign	final DOM power consumption measurements	Due to high risk probability, terminate work on baseline design and switch to new design with lower power density	Additional capEx cost of $40k. Reduced labor required for power monitoring system, some additional labor for power fanout (~40 hrs). Approximately one month schedule delay for timing system and FieldHubs	$ 52,000	$40,000		$65,000	4	3	16		100%	4	$ 52,000.00	0	4	$52,000.00	0
78	TECH45	1.4.3	Because MCA procurement is not finalized, we may have insufficient bandwidth to transmit all data to ICL using the planned cables and comms protocol, which will require additional software or firmware engineering and may reduce science capabilities.	Comms bandwidth	8/18/2018	3/1/2022		Low	Low	Moderate	Low	Low	Moderate	Low	1.6 L2 lead	Until cable procurement is finalized	Mitigate	Inability to identify supplier for cables with planned comms quality	Design protocol for efficiency and robustness, deploy all elements to NTS for integration testing, begin firmware development early if needed; Consider In-OM hit reductions	Add engineering effort - 1 FTE yr - to develop better communications protocol. Development schedule permits completion before deployment	$200,000	$150,000		$300,000	0	0	26		10%	0	$ 200,000.00	Low	0	$20,000.00	Low
79	TECH52	1.4.1.1	Because MCA breakout terminations are a custom solution, the costs of breakout installation may exceed the MSU commitment, requiring project support for MCA costs	Breakout costs	3/28/2022			High	Very low	High	Very low	Low	High	Low	1.4 L2 Lead	Until MCA procurement is finalized: PY4 & PY5	Mitigate	Final bids for breakout installation exceed available MSU funds	Provide cable samples to multiple potential vendors, explore intial designs and negotiate cost reductions with multiple vendors	High cost estimate based on initial bids from two suppliers (JDR and HGS), cost exposure reflects some assumed cost reductions. Schedule risk reflects possibility of extended design phase	$400,000	$0		$750,000	0	0	24		50%	0	$ 400,000.00	0	0	$200,000.00	0
80	TECH46	1.4.1.4	Because the planned YaleGrips have not been demonstrated to work with the planned MCA in a pull test, we may need to increase the YaleGrip tail lengths, which will increase costs	YaleGrip size	2/15/2022	3/1/2022		Low	Very low	Moderate	Very low	Low	Moderate	Low	1.4 L2 Lead	Until final cable design review	Watch	Unsatisfactory pull test with prototype cable	N/A	Replace 461 of YG 944506ET20 quoted @$240 with YG 944507ET22 quoted @$410. Possible longer lead time for custom extended tail grips	$78,000	$75,000		$85,000	0	0	12		20%	0	$ 78,000.00	0	0	$15,600.00	0
81	EXT15	1.4.1.1	Because of supply chain disruptions, we may face delays in production of prototypes and production Main Cables, which will require alternate shipping methods and/or eliminate the option of staging the MCAs at McMurdo the season before installation.	Cable delay	1/15/2021	3/1/2022		Moderate	High	High	Very low	High	High	Low	1.4 L2 Lead	PY4 & PY5	Mitigate	Production is delayed	Offer procurement assistance for long-lead items. Invesitgate and qualify alternative suppliers	Prototypes for qualifying alt. suppliers ~$75k each. Long-lead procurement assistance risks ~$100k. Few-month scale delays still possible	$200,000	$50,000	$225,000	$400,000	26	0	52		25%	26	$ 200,000.00	0	6.5	$50,000.00	0	2021		$180,000	$180,000	$180,000	$0	$0
82	EXT21	1.4	Because there is critical in-kind contributed labor from DESY to design and produce the FieldHub electronics and from Uppsala to design the main cables, the project would be impacted if this labor were lost	Loss of In-Kind Labor (WBS 1.4)	6/27/2022	7/1/2022									1.4 L2 Lead	Until FieldHub shipping	Accept	Loss of key personnel	Complete design and begin production as quickly as possible	Estimate 1 FTE-year of engineering labor, 3 month delay from onboarding for FieldHubs. Most cable work is complete so this is a negligible addition	$ 150,000.00	$ 50,000.00		$ 250,000.00	13	4	26		10%	13	$ 150,000.00	0	1.3	$15,000.00	0
83			1.5 Characterization and Calibration System
84	EXT16	1.5.2.6	Mini Mainboard may be delayed due to components availailability/supply chain issues	MMB delay	1/27/2022	2/28/2022		Moderate	High	Moderate	High	High	Moderate	High	1.5 L2 Lead Williams	2022, calibration module DVT and production	Mitigate	Parts delivery delays/unavailable parts	Obtain key components separately (ahead of time) from turn-key fabrication of Mini-Mainboard assemblies at contract manufacturer.	All ICs have been ordered and are now in hand. Design-specific connectors have either been ordered or are in hand. Passive components have been replaced with components with ample stock.	$40,000	$20,000		$60,000	13	8	52		20%	13	$ 40,000.00	Low	2.6	$8,000.00	Low
85	EXT17	1.5.3.5	IDP is unable to provide a winch for dust logging operations as planned in the baseline.	Dust logging winch availability issue	1/27/2022	2/28/2022		Moderate	Low	Moderate	Moderate	Moderate	Moderate	Moderate	NSF	2022 discussions with IDP, 2024 testing	Early communications	Approval delay at IDP, lack of engineering support at IDP	Working with IDP to secure the loan of this equipment well in advance of the date needed to allow them ample time to prepare the unit for use. In parallel, ICU is staying in touch with vendors to understand lead times should a winch procurement be required.	Risk is retired after testing winch with logger (Spring/Summer 2024). Cost is of purchase of a plausible additional winch if loan is unavailable.	$145,000	$140,000		$160,000	8	6	10		20%	8	$ 145,000.00	Moderate	1.6	$29,000.00	Low
86	EXT22	1.5	Because there is critical in-kind contributed labor from the IceCube M&O effort as well as the stand alone calibration devices which are delivered by a variety of external partners, the project would be impacted if this labor were lost	Loss of In-Kind Labor (WBS 1.5)	6/27/2022										L2	Until OM shipping	Accept	Loss of key personnel		Critical labor for calibration devices and analysis of the initial hole ice is in-kind, if it is not available then there will need to be an additional on-project labor estimate, here estimated as 1+/-20% FTE.	$ 100,000.00	$ 80,000.00		$ 120,000.00	12	4	24		15%	12	$ 100,000.00	Low	1.8	$15,000.00	Low
87			1.6 M&O Data Systems Integration
88	TECH47	1.6.1.1	If optical sensor or calibration device software interfaces specifications are delayed will impact DAQ software development	DAQ software interface delays	8/17/2018			Low	Moderate	Low	Low	Moderate	Low	Low	1.6 L2 lead	Sensor software delivery milestones	Better communications	Delivery delays	Work closely with Sensor and Calibration devices L2s to ensure software interfaces are developed and adhered to. Interface specification workshop	Minor cost impact, mostly delays to schedule. Estimate 20 week possible delay, and 0.1 FTE*yr	$20,000	$0	$20,000	$40,000	20	10	30		25%	20	$ 20,000.00	0	5	$5,000.00	0
89	TECH48	1.6.1.1	Because the hit rate from OM devices exceeds the rate available in the DAQ trigger system, and triggers/hits could potentially be missed and physics events missed.	DAQ trigger too high	8/17/2018	1/25/2022 (increased probability)		Moderate	Low	Moderate	Moderate	Moderate	Moderate	Moderate	1.6 L2 lead	DAQ software testing	Mitigate	DAQ trigger testing failures	Refactor DAQ trigger algorithms to allow for higher rates, including developing noise rejection methods or look to reduce rate of trigger primitives from modules via cuts	Would require potential refactoring of DAQ architecture to support additional rates. If realized, could be an 0.5FTE*yr and 20 week delay	$80,000	$40,000	$80,000	$120,000	20	10	30		5%	20	$ 80,000.00	Low (~5% or less of physics hits expected to be impacted by noise cleaning	1	$4,000.00	Low
90	TECH49	1.6.1.1	If the per-PMT noise rates in mDOMs exceed design specifications, the in-OM data processing and noise mitigation algorithms may not have enough in-OM resources and data could potentially be lost.	mDOM noise rates overwhelm MCU	1/25/2022			Moderate	Low	Low	Moderate	Moderate	Moderate	Moderate	1.6 L2 lead	Integrated mDOM testing	Mitigate	MCU can't keep up with rate of mDOM PMTs	Use FPGA firmware resources to offload MCU; develop more CPU efficient feature extraction algorithms	Minimal cost impacts or schedule impacts: these mitigations are already planned, just need to retune and understand physics impact; 0.1 FTE*yr and 10 week schedule impact potential	$20,000	$0	$15,000	$30,000	10	5	15		5%	10	$ 20,000.00	Low (potentially small losses in physics hit fidelity due to reducing CPU/FPGA computing impacts	0.5	$1,000.00	Low
91	TECH50	1.6.1.1	If an in-ice device could be identified as the wrong type at boot up, improper hardware configuration could be used, leading to damage of components.	MCU SW Config ID failure	1/25/2022			Low	Very Low	Low	Low	Low	Low	Low	1.6 L2 lead	FAT testing of modules	Mitigate	Module type incorrectly ID by HW selection and hardware systems potentially damaged	Ensure robust testing of all mainboards as early as possible to ID issues with this key circuit; implement additional checks against backup IDs in scratch nvram	Addiitional internal software safety checks might be needed. 0.25 FTE*yr, 6 weeks schedule	$40,000	$20,000	$40,000	$60,000	6	3	9		3%	6	$ 40,000.00	Very low - safety checks to avoid any potential damage	0.18	$1,200.00	Low
92	TECH51	1.6.1.1	If the total data rate from all connected devices exceeds the total available processing bandwidth, the FieldHub will be unable to keep up with data transfer from all connected devices and data could be lost.	FieldHub software unable to keep up	1/25/2022			Low	Low	Low	Low	Low	Low	Low	1.6 L2 lead	FieldHub full scale testing at NTS	Mitigate	Fully loaded field hub can't process all hits from ~100 connected DOMs	Scale tests on pre-production field hub units, investigate non-USB transfer mechanisms, parallelism in SBC design.	Potentially need to investigate alternative data transfer methods internally, or additional parallelism. Work generally off-Upgrade funding, but potential if M&O resources overwehlmed.	$25,000	$0	$20,000	$40,000	20	5	30		5%	20	$ 25,000.00	0	1	$1,250.00	0
93	EXT23	1.6	Because there is critical in-kind contributed labor from the IceCube M&O project in the firmware and software for the Upgrade, the project would be impacted if this labor were lost	Loss of In-Kind Labor (WBS 1.6)	6/27/2022										1.6 L2 lead	Until string and software deployment	Accept	Loss of key personnel	Expand software and firmware expertise among as large of a group as possible	Development of several key sofware and firmware systems are planned for delivery from M&O support people. These people are supported by NSF grants, which are not gauranteed	$ 200,000.00	$ 50,000.00		$ 300,000.00	26	0	40		25%	26	$ 200,000.00	low	6.5	$50,000.00	Low
94			Opportunities
95	OPP1	1.1	If contributed drillers (from collaborating institutions) are provided as in-kind contributions, the project can save seasonal driller labor costs.	Contributed drillers	3/8/2022			Moderate	Very low	High	Very low	Low	High	Low	1.1 PM	Ahead of FS1	Opportunity	Volunteers from the collaboration	Will advertise it widely within the collaboration. Fixed FS years will help significantly.	Contributed drillers (up to 10 person*seasons) replace $50000 in direct costs.	$250,000	$100,000		$500,000	0	0	0		50%	0	$ 250,000.00	Very low	0	$125,000.00	Low
96
97			Retirement home + Deleted risks
98	RET1	1.6.4	If there are delays in design of the IDF, computing infrastructure equipment ordering and shipping could be delayed - retire	IDF late design	8/17/2018	7/4/2019	7/4/2019	Low	Low	Moderate	Very low	Low	Moderate	Low	1.6 L2 lead	IDF facility design delivery	Mitigate	IDF Design delays	Separate network/computing design elements in the IDF design process, preventing delays in other areas from largely impacting 1.6	Minor cost impact. The IFD is no longer being used in the IceCube Upgrade Project.			$0						Low	Low	Low	Very low	Low	Low	Low
99	RET2	1.6.1.1	If optical sensor or calibration device software interfaces specifications are delayed will impact DAQ software development Includes ICM Firmware Development. This includes the delivery of the CPT infrastructure. (Combined Risks TECH15 and TECH16) COVID-19	DAQ software interface delays Risks: 3, 10	8/17/2018		1/25/2022	Low	Moderate	Very low	Very low	high	Low	Low	1.6 L2 lead	Sensor software delivery milestones	Better communications	Delivery delays	Work closely with Sensor and Calibration devices L2s to ensure software interfaces are developed and adhered to. Interface specification workshop. Prioritize remote development efforts, such as FAT software readiness using networked main boards.										high	moderate	low	low	High	Moderate	Moderate
100	RET3	1.6.1	If there are delays in delivery of CPT infrastructure for testing (Field Hubs, SJBs, NTS) then the development of Online Systems will be delayed	CPT milestone delays	8/17/2018		1/25/2022	Low	Moderate	Very low	Very low	Moderate	Low	Low	1.6 L2 lead	CPT Hardware delivery date	Better communications	Delivery delays	Work closely with CPT L2 and L3s to ensure at least partially hardware delivery for development and testing	Minor			$0						Low	Low	Very low	Very low	Low	Low	Low

Back to top

---