Renewal of the Lease for the NYCERS Disaster Recovery Site at Long Island City

In late spring of 2006 NYCERS signed a ten year lease for a disaster recovery site in Long Island City. It comes up for renewal this year.

As of today NYCERS still does not have a Certificate of Occupancy for the site. This means that there has never been a full fledged test of a recovery plan since legally workers are not permitted on the site. I have no idea how NYCERS can safely run their file backup system at this site when there is no permanent staff at the site. As far as I know there is no plan B.

NYCERS has an option to renew the lease for up to two five year periods.

What do you think D'Alessandro will do? What do you think the trustees will do?

The Haze Over the Disaster Recovery Site at Long Island City - 2013

On March 14, 2013 the NYCERs Board of Trustees adopted resolutions approving the NYCERS administrative and loan operating budgets for FY-2014. The administrative budget was $44.6M with an added $7.6M for fringe benefits. The loan budget was $1.7M with $.3M for fringe benefits. The loan budget is totally paid by the active members through fees for loans. The executive director, Diane D’Alessandro, provided a two and a half page justification for these expenditures.

2011

In September 12, 2011 posting, I wrote about the fiasco surrounding the NYCERS disaster recovery (DR) site at Long Island City (LIC). At the time there was a five year delay in opening the DR site.

2012

In her February, 2012 justification for the FY-2013 NYCERS administrative budget D’Alessandro referred to the DR site issue as follows:

“The build-out of the business continuity site is nearing completion. Once the construction phase is finalized, the installation of the secondary data center will commence. This final phase will extend through FY-2103. Upon completion, the secondary data center will be equipped with a mainframe computer and have the capability for site-to-site data replication and data updating allowing for the eventual elimination of the Sterling Forest backup site. In an emergency, the LIC will have the capacity to house approximately 300 staff working on a two shift model. During normal business, the facility will be utilized for document management, training and IT testing and backup functions.“

In a July 13, 2012 posting, I commented on D’Alessandro’s failure to address the then six year delay in opening the LIC DR site and all the other problems surrounding this project.

2013

Part of the FY-2014 administrative budget is a $2.9M non payroll (OTPS) cost for maintaining the LIC DR site. In her FY-2014 budget justification D’Alessandro makes only the following mention of the LIC site:

“With the completion of the secondary data center at Long Island City (LIC), NYCERS will shift its focus to restructuring the Adams Street data center. Proposed upgrades will improve the utilization of space and enhance cooling, heating and air flow systems. This effort will provide greater energy efficiency, expanded storage capacity and improved functionality.”

From this terse comment you might assume that the DR site is fully functional. You will notice, however, that there is no mention of the actual completion date of the site, the award of a certificate of occupancy or the passage of fire inspection for the LIC site. There is also no mention of maximum occupancy for the site and there is no mention of a full operational test of the site. Remember the original 2006 lease renewal comes up in FY-2016, just two years away.

Given the history of this project it is quite possible that the site is not operational at all in spite of D’Alessandro’s attempt to give that impression. Of course it now seems that replacing the air conditioning unit in the computer room at Adams Street is the new hot item. At least, it appears that is what she is talking about. "Restructuring" and "Upgrades" are not words loaded with a lot of information.

Disaster Recovery and Hurricane Sandy

With Hurricane Sandy’s knockout of major office buildings in lower Manhattan, metro area organizations will all be reviewing the effectiveness of their disaster recovery plans.

In 2004, I began the effort to establish an alternative disaster recovery office site for NYCERS. It is unclear, eight years later, whether the NYCERS Long Island City recovery site would actually have been able to function as an effective disaster recovery site, if Hurricane Sandy had flooded 335 Adams Street as it did 55 Water Street, the home of the NYC Teachers Retirement System.

Background Update - Disaster Recovery Site: Has NYCERS Fixed the Problem?

Update - July 24, 2018

While doing some research on the current "Legacy Replacement" fiasco, I came across an item in the FY-2012 NYCERS financial statement (CAFR). It was for the cost of the build out for the Long Island City (LIC) disaster recovery site. That cost was $3,791,813 paid out as of June 30, 2012. You will never hear any NYCERS staff mention this number in conjunction with LIC.

End of update.

Original post - July 13. 2012

Last week I received a copy of the NYCERS Administrative Budget for FY-2013.

In September, 2011, I commented on the five year delay in completing the NYCERS Long Island City (LIC) disaster recovery (DR) site.

Based on the NYCERS budget proposal for 2012-2013, the site is now “nearing” completion with a date sometime before June 30, 2013.

There continues to be an ongoing confusion about the completion of this project. On March 25, 2012 in response to a newspaper article about the five year delay in getting the LIC site operational Karen Mazza, the NYCERS legal director, stated that the site would be ready to open on April 1, 2012. Considering the history of this project we’ll need to see the certificate of occupancy before being sure that the site is open. Of course, having a fully functioning alternative site for all agency operations is a whole other story.

The rent for the LIC site for the coming year is $496,500.

In its FY-2013 budget NYCERS’s has increased its headcount to a total of 394 F/T and 25 P/T positions. In addition there are always a significant number of consultants working at NYCERS.

As per the budget document the DR site is projected to only support 150 people at any one time or 300 people using a two shift schedule. The certificate of occupancy and the FDNY inspection will actually determine the working capacity of the site.

The two shift schedule raises contract issues with the unions involved and creates serious hardships for employees stuck with off hours schedules. It also leaves a problem for the remaining 94 employees and consultants and their associated work.

Part of the long delayed plan is to install a second mainframe at LIC with the idea of ending the use of the IBM DR site at Sterling Forest, NY.

This is fundamentally a bad idea. The elimination of the Sterling Forest site means that NYCERS will lose flexibility in its ability to respond to unknown disasters. It also creates personnel, management, and cost challenges in keeping the LIC mainframe in synch with the Adams Street machine. NYCERS is a relatively small organization which needs to avoid high overhead and to leverage its skills with the use of the Sterling Forest site.

Prior to 2005 NYCERS had an IT expert as the executive director. The current executive director has no business experience prior to 2005. This makes NYCERS very susceptible to “pie in the sky” schemes as evidenced by the six year delay in getting this project off the ground.

As per the budget document, NYCERS is also planning to move its document management, training, and IT testing/backup functions to this site.

It is not clear what NYCERS will do with its modern training center at Adams Street. Adams Street is the site where all NYCERS employees are currently working. It is clearly a more efficient site to train employees than LIC.

Moving document management to Long Island City will only create delays in agency functions and increase the difficulty in controlling documents that arrive at Adams Street and then will have to be transported to LIC site for processing.

The current offsite record storage at Bergen Street will be moved to the DR site. This was a pre-2005 concept. I wonder how it survived.

What do you get for $2.9M?

A notice in the October 31, 2011 issue of the City Record indicated that NYCERS had award a contract to the Caudwell Wingate Company, LLC for the demolition and build-out of a disaster recovery site. The amount of the award was $2,897,271. This was a negotiated acquisition.

There is a problem with this contract. NYCERS does not have a budget authorization for this $2.9M item in its FY-2012 budget. The trustees will need to modify the FY-2012 budget to cover this expense. As of now, the NYCERS executive director does not have the authority to sign this contract unless the trustees have authorized the funds needed to pay for it.

What is NYCERS supposed to be getting for this $2.9M? In April of this year, the NYCERS legal department described the work as follows: “ to include but not limited to demolition of existing walls, installation of walls, HVAC, lighting, flooring, electrical, millwork, new operable partition wall and sink in pantry”. That must be some sink. This work is being done on a single floor in an old warehouse site where a retail firm used to store some of its inventory. In passing, it appears that the NYCERS legal department has been reassigned to procurement and site management.

When is this project going to be completed? Mañana.

Who is Caudwell Wingate and what are their political connections? This week the NY Times had an interesting story about a boiler installation contract at the new Yankee Stadium.

Considering that NYCERS signed the lease for the disaster recovery site in L.I.C. in 2006 and it is still not yet operational, the trustees should be closely monitoring everything involved with this project.

Everyone is screaming about a pension crisis. This is the type of management failure that causes a crisis.

Phantom Disaster Recovery Site at Long Island City

In 1997, after NYCERS received independent budgeting authority, we began developing a computer disaster recovery plan. In conjunction with IBM and its Sterling Forest site we constructed a process where we were able restore our computer systems at Sterling Forest along with our key operations. This plan only supported about 50 operations employees.

In February, 2000, we moved NYCERS from 220 Church St. to 335 Adams Street in Brooklyn. This new office site had 3 general office floors (21 to 23) and a mezzanine site where we located our customer service center, our call center, and medical board division for handling disabilities.

After of 9/11, everyone in the New York metropolitan area became very aware of the potential for wide spread disasters.

In early 2004, NYCERS experienced two minor business interruptions at our new office site. One was a broken sprinkler pipe on the 24th floor and the other was a fire in an electrical closet on the 8th floor. The water damage was minor but the fire closed the 3 main office floors for a week. We were lucky that the mezzanine still had power. We had installed a second data line from Sterling Forest to the mezzanine along with the main data line on the 23th floor. We also had installed a backup PBX in the mezzanine which allowed the call center to operate after two days.

While the recovery was a great success, it was clear to me that any long term outage would be unacceptable with respect to agency’s work load and the management our staff of 370 employees.

I proposed to the trustees as part of the FY-2005 budget package that NYCERS should explore a possible alternate work site in the event of a local disaster at 335 Adams Street. We would be looking for site within the city, on a different power grid, with decent mass transit access, capable of housing at least 250 workers at one time, capable of being linked up to Sterling Forest, 335 Adams Street, and other city data centers, and finally not too expensive.

The site would have to have the potential to be fully outfitted but we would start with only a minimal kernel for immediate operations. This would include a full technology infrastructure. We would slowly install non-critical equipment over time. In the event of a disaster, we would expedite that process. The site would have the added advantage of allowing us more frequent and realistic disaster testing.

I was looking for a 5 year lease with an option to renew. I was hoping the city would have developed a local DR site for general city agencies within that time frame. That was wishful thinking. I had told the trustees that after we had identified a site, I would present them with a total cost proposal and plan for their final approval.

The Deputy Director of Administration, Niki Browne, was in charge of the search. Late in 2004 Ms. Browne identified a potential site in Long Island City. It was, however, warehouse space and had many issues to be resolved. Unfortunately in June of 2005, Ms. Browne was removed from the project and it was turned over to Karen Mazza. Needless to say, this project is above her pay grade. In fact, Mazza, a lawyer, has serious problems with filling out time sheets, handling her emails, and analyzing legislation.

In October, 2005, Martha Stark and the trustees appointed Diane D’Alessandro as the new executive director of NYCERS.

On of May 31, 2006, D’Alessandro signed a lease with 30TH Place Holdings, LLC for space on the 10th floor of The Factory Building, 30-00 47th Avenue, Long Island City, NY (LIC). This was the site that Ms. Browne had originally identified. The term of the lease was ten years with an option to renew for 5 years. The dimensions of the space were not specified in the lease. NYCERS began paying rent circa 2/1/2007 ($141,905 for FY-2007).

The annual rents for the ten years are listed at the end of this posting. These amounts do not include taxes, HLtP, or water charges.

In her February 25, 2011 letter to the trustees presenting the FY-2012 budget, D’Alessandro notified the trustees that the development of the disaster recovery facility was continuing. She also stated that the LIC site will have the capacity to house two shifts of 150 people. NYCERS currently has 396 employees and a large number of consultants.

I have recently commented on the fact that as of this past summer basic construction of the LIC site had not yet been started. This is over five years since the lease was signed.

D’Alessandro makes no mention to the trustees of this enormous delay in getting the site operational. She also does not mention the fact that, even if the site is made operational, it will not be able to handle the full current workforce. This is beginning to fit the CityTime pattern as far as the loss of time and money. NYCERS has budgeted over $8.7M from FY-2007 to FY-2012 for the LIC site.

NYCERS actually spent over $2.2M between FY02007 and FY-2009 on the LIC site. As of FY-2010, NYCERS stopped publicly reporting the ongoing rent payments and the associated costs for the LIC site.

This lack of disclosure is not unique to the LIC site. It applies to all NYCERS paid administrative expenses. The trustees no longer have any way of checking on actual spending at NYCERS. You can be sure the Comptroller is not auditing the financial controls at NYCERS. He has enough trouble managing the investment expenses for the pension funds. And, I wouldn’t count on the outside accounting firm either.

My professional opinion is this site is seriously flawed and will never be able to function at any acceptable level as a business disaster recovery site for NYCERS. I suspect that at the time of the lease signing NYCERS had not solved all the operational problems connected to the site and that some of those problems are intractable. Let’s hope NYCERS has the good sense not to extend the lease in 2016.

Annual rents for each year of the ten year lease for 30-00 47th Ave, LIC, NY:

1. $362,076
2. $370,222
3. $378,552
4. $387,070
5. $479,335
6. $490,120
7. $501,148
8. $512,423
9. $523,953
10. $535,742

Five Years Lost on Disaster Recovery Site

Five years ago (FY-2007) NYCERS signed a lease for a site in Long Island City to be used as a backup site for NYCERS in case of disaster at its downtown Brooklyn office.The annual rent is approximately is $375,000.

On June 27, 2011 Karen Mazza, a staff attorney at NYCERS placed a notice in the City Record that NYCERS was soliciting a negotiated acquisition for the demolition and build-out of a disaster recovery data center. The due date is July 8, 2011, a 12 day turnaround. Don't hold your breath.

It took almost five years for NYCERS to move from the lease signing to the selection of contractor to build the data center at the disaster recovery site. In order for the site to be a step up from the current IBM Sterling Forest D.R. program, the site will also have to be built out to accommodate at least 250 employees working alternate shifts. This would allow NYCERS to continue business operations for extend periods of time (i.e.: six months) in the event of a disaster.

It is clear that NYCERS has already wasted $1.6M on lease payments alone, not counting any other associated costs. That money came from contributions from members, retirees, and employers who are stakeholders in the pension fund. It is reasonable to assume that this current effort will fail as has efforts since FY-2007.

As of FY-2010 NYCERS has stopped reporting a detailed reconciliation of budgeted and actual expenses to the trustees. Needless to say, the trustees don't seem to care.