Renewal of the Lease for the NYCERS Disaster Recovery Site at Long Island City

In late spring of 2006 NYCERS signed a ten year lease for a disaster recovery site in Long Island City. It comes up for renewal this year.

As of today NYCERS still does not have a Certificate of Occupancy for the site. This means that there has never been a full fledged test of a recovery plan since legally workers are not permitted on the site. I have no idea how NYCERS can safely run their file backup system at this site when there is no permanent staff at the site. As far as I know there is no plan B.

NYCERS has an option to renew the lease for up to two five year periods.

What do you think D'Alessandro will do? What do you think the trustees will do?

More Haze Over Long Island City

In September I wrote about the adoption of the NYCERS budget for FY-2014 at the March 14, 2013 NYCERS Board of Trustees meeting. I pointed out the scant reference to the issues surrounding the disaster recovery site at Long Island City (L.I.C.) in the executive director's written presentation on the budget. In a terse sentence she implied that the L.I.C. disaster recovery project was now functioning after an seven year delay.

I was curious whether any of the trustees had any questions at the meeting about this fiasco. So I made a FOIL request of the public minutes of the board meeting where the $52.2M budget was approved. This is the relevant exert form the minutes of the March 14, 2013 NYCERS Board of Trustees regular meeting:

CHAIRPERSON WOLPERT:
All in favor?
(A chorus of "Ayes.")
Any opposed?
Motion carries.
Item 2 on the agenda is a report by the budget committee, of which I am chair, so I will report on that.

I think NYCERS really did an incredible job presenting their budget to us. There was beautiful and substantively informative PowerPoint presentation. All of the really knowledgeable staff was there to answer any questions we had. For me, I think everything was in order.

I don't know if anybody on the budget committee or anybody else has any questions?

MS. O'CONNELL (DC-37):
A comment to echo your comments, Carolyn. I think Diane and staff did an excellent job of keeping the focus on member services, and also moving the institution forward in terms of technology and making progress in that regard. So I thank the executive director and staff for their efforts.

CHAIRPERSON WOLPERT:
With that, a motion to approve the budget?

MS. D"ALESSANDRO(executive director):
...Then we move into the actual proposed budget for fiscal 2014 which is 2B(i) in your packet, and essentially it is the administrative budget for fiscal 2014 with $26,620,635 for PS expenses; $17,951,822 for other than personal expenses; qnd a total budget of $44,575,457 for fiscal 2014. (there is an additional $7.6M in fringe costs)

CHAIRPERSON WOLPERT:
Any questions on this resolution?
Is there a motion to approve?

MS. O'CONNELL:
Motion.

MS. STRYKER(Local 237, Teamsters):
Second.

CHAIRPERSON WOLPERT:
All in favor?
(A chorus of "Ayes.")
Any opposed?
Motion carries.

As you can see from the quoted text above, that there was no substantive discussion about this major expenditure. The text is devoid of any concrete information other than the fact that the regular NYCERS admin budget for FY-2014 will be $44.5M. (actually $52.2M)

One thing that caught my attention was the comment about the "beautiful and substantively informative PowerPoint presentation". I thought that it would be wonderful to read the presentation given to the budget committee. So I made a follow up FOIL request for it. That was on October 18, 2013. I made a second request on November 11, 2013 and on December 3, 2013 I received the following response from NYCERS.

Please be advised that your request under the Freedom of Information Law for NYCERS PowerPoint presentation outlining the FY-2014 administrative budget given to the Trustees budget committee is being denied.

The information you are seeking is considered inter-agency or intra-agency materials. Said materials were not presented to the Board of Trustees or listed as an Agenda Item at any NYCERS Regular/Investment meeting.

If you wish to appeal this denial, please contact Diane D’Alessandro, (Executive Director) of NYCERS.

It is clear to me that this reason for denial is evasive and incorrect. Listed below is the exact wording from the state web site about this specific reason for denying a FOIL request:

(g) are inter-agency or intra-agency communications, except to the extent that such materials consist of:
i. statistical or factual tabulations or data;
ii. instructions to staff that affect the public;
iii. final agency policy or determinations; or
iv. external audits, including but not limited to audits performed by the comptroller and the federal government;

As per the meeting minutes, the Chairperson points to the PowerPoint presentation as the basis of the budget committee's recommendation for approval of the proposed budget.I think it is safe to say the document was the basis of a final agency policy or determination, the adoption of the agency's $52.2M FY-2014 administrative budget.

In addition, I suspect that the Budget Committee falls under the N.Y.S. Open Meetings Law. That means material provided to the committee is public record. I don't think the committee had any reason to conduct its meeting in executive session.

Needless to say, while I will never see this PowerPoint presentation, I am sure that the N.Y.S. Department of Financial Services auditors will demand to see this presentation. They should also demand a physical inspection of the L.I.C. site and observe actual full disaster drill, not some empty "trust me" claim.

You would think that NYCERS management would have been anxious to provide the public a clear picture of the status of the Long Island City disaster recovery site. Now, because of this failure to disclose, it seems reasonable to conclude that the PowerPoint presentation is inaccurate and a poor basis for adopting the FY-2014 budget. Besides the mess at Long Island City what other disasters are the NYCERS management hiding.

The Haze Over the Disaster Recovery Site at Long Island City - 2013

On March 14, 2013 the NYCERs Board of Trustees adopted resolutions approving the NYCERS administrative and loan operating budgets for FY-2014. The administrative budget was $44.6M with an added $7.6M for fringe benefits. The loan budget was $1.7M with $.3M for fringe benefits. The loan budget is totally paid by the active members through fees for loans. The executive director, Diane D’Alessandro, provided a two and a half page justification for these expenditures.

2011

In September 12, 2011 posting, I wrote about the fiasco surrounding the NYCERS disaster recovery (DR) site at Long Island City (LIC). At the time there was a five year delay in opening the DR site.

2012

In her February, 2012 justification for the FY-2013 NYCERS administrative budget D’Alessandro referred to the DR site issue as follows:

“The build-out of the business continuity site is nearing completion. Once the construction phase is finalized, the installation of the secondary data center will commence. This final phase will extend through FY-2103. Upon completion, the secondary data center will be equipped with a mainframe computer and have the capability for site-to-site data replication and data updating allowing for the eventual elimination of the Sterling Forest backup site. In an emergency, the LIC will have the capacity to house approximately 300 staff working on a two shift model. During normal business, the facility will be utilized for document management, training and IT testing and backup functions.“

In a July 13, 2012 posting, I commented on D’Alessandro’s failure to address the then six year delay in opening the LIC DR site and all the other problems surrounding this project.

2013

Part of the FY-2014 administrative budget is a $2.9M non payroll (OTPS) cost for maintaining the LIC DR site. In her FY-2014 budget justification D’Alessandro makes only the following mention of the LIC site:

“With the completion of the secondary data center at Long Island City (LIC), NYCERS will shift its focus to restructuring the Adams Street data center. Proposed upgrades will improve the utilization of space and enhance cooling, heating and air flow systems. This effort will provide greater energy efficiency, expanded storage capacity and improved functionality.”

From this terse comment you might assume that the DR site is fully functional. You will notice, however, that there is no mention of the actual completion date of the site, the award of a certificate of occupancy or the passage of fire inspection for the LIC site. There is also no mention of maximum occupancy for the site and there is no mention of a full operational test of the site. Remember the original 2006 lease renewal comes up in FY-2016, just two years away.

Given the history of this project it is quite possible that the site is not operational at all in spite of D’Alessandro’s attempt to give that impression. Of course it now seems that replacing the air conditioning unit in the computer room at Adams Street is the new hot item. At least, it appears that is what she is talking about. "Restructuring" and "Upgrades" are not words loaded with a lot of information.

Disaster Recovery and Hurricane Sandy

With Hurricane Sandy’s knockout of major office buildings in lower Manhattan, metro area organizations will all be reviewing the effectiveness of their disaster recovery plans.

In 2004, I began the effort to establish an alternative disaster recovery office site for NYCERS. It is unclear, eight years later, whether the NYCERS Long Island City recovery site would actually have been able to function as an effective disaster recovery site, if Hurricane Sandy had flooded 335 Adams Street as it did 55 Water Street, the home of the NYC Teachers Retirement System.

Background Update - Disaster Recovery Site: Has NYCERS Fixed the Problem?

Update - July 24, 2018

While doing some research on the current "Legacy Replacement" fiasco, I came across an item in the FY-2012 NYCERS financial statement (CAFR). It was for the cost of the build out for the Long Island City (LIC) disaster recovery site. That cost was $3,791,813 paid out as of June 30, 2012. You will never hear any NYCERS staff mention this number in conjunction with LIC.

End of update.

Original post - July 13. 2012

Last week I received a copy of the NYCERS Administrative Budget for FY-2013.

In September, 2011, I commented on the five year delay in completing the NYCERS Long Island City (LIC) disaster recovery (DR) site.

Based on the NYCERS budget proposal for 2012-2013, the site is now “nearing” completion with a date sometime before June 30, 2013.

There continues to be an ongoing confusion about the completion of this project. On March 25, 2012 in response to a newspaper article about the five year delay in getting the LIC site operational Karen Mazza, the NYCERS legal director, stated that the site would be ready to open on April 1, 2012. Considering the history of this project we’ll need to see the certificate of occupancy before being sure that the site is open. Of course, having a fully functioning alternative site for all agency operations is a whole other story.

The rent for the LIC site for the coming year is $496,500.

In its FY-2013 budget NYCERS’s has increased its headcount to a total of 394 F/T and 25 P/T positions. In addition there are always a significant number of consultants working at NYCERS.

As per the budget document the DR site is projected to only support 150 people at any one time or 300 people using a two shift schedule. The certificate of occupancy and the FDNY inspection will actually determine the working capacity of the site.

The two shift schedule raises contract issues with the unions involved and creates serious hardships for employees stuck with off hours schedules. It also leaves a problem for the remaining 94 employees and consultants and their associated work.

Part of the long delayed plan is to install a second mainframe at LIC with the idea of ending the use of the IBM DR site at Sterling Forest, NY.

This is fundamentally a bad idea. The elimination of the Sterling Forest site means that NYCERS will lose flexibility in its ability to respond to unknown disasters. It also creates personnel, management, and cost challenges in keeping the LIC mainframe in synch with the Adams Street machine. NYCERS is a relatively small organization which needs to avoid high overhead and to leverage its skills with the use of the Sterling Forest site.

Prior to 2005 NYCERS had an IT expert as the executive director. The current executive director has no business experience prior to 2005. This makes NYCERS very susceptible to “pie in the sky” schemes as evidenced by the six year delay in getting this project off the ground.

As per the budget document, NYCERS is also planning to move its document management, training, and IT testing/backup functions to this site.

It is not clear what NYCERS will do with its modern training center at Adams Street. Adams Street is the site where all NYCERS employees are currently working. It is clearly a more efficient site to train employees than LIC.

Moving document management to Long Island City will only create delays in agency functions and increase the difficulty in controlling documents that arrive at Adams Street and then will have to be transported to LIC site for processing.

The current offsite record storage at Bergen Street will be moved to the DR site. This was a pre-2005 concept. I wonder how it survived.

What do you get for $2.9M?

A notice in the October 31, 2011 issue of the City Record indicated that NYCERS had award a contract to the Caudwell Wingate Company, LLC for the demolition and build-out of a disaster recovery site. The amount of the award was $2,897,271. This was a negotiated acquisition.

There is a problem with this contract. NYCERS does not have a budget authorization for this $2.9M item in its FY-2012 budget. The trustees will need to modify the FY-2012 budget to cover this expense. As of now, the NYCERS executive director does not have the authority to sign this contract unless the trustees have authorized the funds needed to pay for it.

What is NYCERS supposed to be getting for this $2.9M? In April of this year, the NYCERS legal department described the work as follows: “ to include but not limited to demolition of existing walls, installation of walls, HVAC, lighting, flooring, electrical, millwork, new operable partition wall and sink in pantry”. That must be some sink. This work is being done on a single floor in an old warehouse site where a retail firm used to store some of its inventory. In passing, it appears that the NYCERS legal department has been reassigned to procurement and site management.

When is this project going to be completed? Mañana.

Who is Caudwell Wingate and what are their political connections? This week the NY Times had an interesting story about a boiler installation contract at the new Yankee Stadium.

Considering that NYCERS signed the lease for the disaster recovery site in L.I.C. in 2006 and it is still not yet operational, the trustees should be closely monitoring everything involved with this project.

Everyone is screaming about a pension crisis. This is the type of management failure that causes a crisis.

Phantom Disaster Recovery Site at Long Island City

In 1997, after NYCERS received independent budgeting authority, we began developing a computer disaster recovery plan. In conjunction with IBM and its Sterling Forest site we constructed a process where we were able restore our computer systems at Sterling Forest along with our key operations. This plan only supported about 50 operations employees.

In February, 2000, we moved NYCERS from 220 Church St. to 335 Adams Street in Brooklyn. This new office site had 3 general office floors (21 to 23) and a mezzanine site where we located our customer service center, our call center, and medical board division for handling disabilities.

After of 9/11, everyone in the New York metropolitan area became very aware of the potential for wide spread disasters.

In early 2004, NYCERS experienced two minor business interruptions at our new office site. One was a broken sprinkler pipe on the 24th floor and the other was a fire in an electrical closet on the 8th floor. The water damage was minor but the fire closed the 3 main office floors for a week. We were lucky that the mezzanine still had power. We had installed a second data line from Sterling Forest to the mezzanine along with the main data line on the 23th floor. We also had installed a backup PBX in the mezzanine which allowed the call center to operate after two days.

While the recovery was a great success, it was clear to me that any long term outage would be unacceptable with respect to agency’s work load and the management our staff of 370 employees.

I proposed to the trustees as part of the FY-2005 budget package that NYCERS should explore a possible alternate work site in the event of a local disaster at 335 Adams Street. We would be looking for site within the city, on a different power grid, with decent mass transit access, capable of housing at least 250 workers at one time, capable of being linked up to Sterling Forest, 335 Adams Street, and other city data centers, and finally not too expensive.

The site would have to have the potential to be fully outfitted but we would start with only a minimal kernel for immediate operations. This would include a full technology infrastructure. We would slowly install non-critical equipment over time. In the event of a disaster, we would expedite that process. The site would have the added advantage of allowing us more frequent and realistic disaster testing.

I was looking for a 5 year lease with an option to renew. I was hoping the city would have developed a local DR site for general city agencies within that time frame. That was wishful thinking. I had told the trustees that after we had identified a site, I would present them with a total cost proposal and plan for their final approval.

The Deputy Director of Administration, Niki Browne, was in charge of the search. Late in 2004 Ms. Browne identified a potential site in Long Island City. It was, however, warehouse space and had many issues to be resolved. Unfortunately in June of 2005, Ms. Browne was removed from the project and it was turned over to Karen Mazza. Needless to say, this project is above her pay grade. In fact, Mazza, a lawyer, has serious problems with filling out time sheets, handling her emails, and analyzing legislation.

In October, 2005, Martha Stark and the trustees appointed Diane D’Alessandro as the new executive director of NYCERS.

On of May 31, 2006, D’Alessandro signed a lease with 30TH Place Holdings, LLC for space on the 10th floor of The Factory Building, 30-00 47th Avenue, Long Island City, NY (LIC). This was the site that Ms. Browne had originally identified. The term of the lease was ten years with an option to renew for 5 years. The dimensions of the space were not specified in the lease. NYCERS began paying rent circa 2/1/2007 ($141,905 for FY-2007).

The annual rents for the ten years are listed at the end of this posting. These amounts do not include taxes, HLtP, or water charges.

In her February 25, 2011 letter to the trustees presenting the FY-2012 budget, D’Alessandro notified the trustees that the development of the disaster recovery facility was continuing. She also stated that the LIC site will have the capacity to house two shifts of 150 people. NYCERS currently has 396 employees and a large number of consultants.

I have recently commented on the fact that as of this past summer basic construction of the LIC site had not yet been started. This is over five years since the lease was signed.

D’Alessandro makes no mention to the trustees of this enormous delay in getting the site operational. She also does not mention the fact that, even if the site is made operational, it will not be able to handle the full current workforce. This is beginning to fit the CityTime pattern as far as the loss of time and money. NYCERS has budgeted over $8.7M from FY-2007 to FY-2012 for the LIC site.

NYCERS actually spent over $2.2M between FY02007 and FY-2009 on the LIC site. As of FY-2010, NYCERS stopped publicly reporting the ongoing rent payments and the associated costs for the LIC site.

This lack of disclosure is not unique to the LIC site. It applies to all NYCERS paid administrative expenses. The trustees no longer have any way of checking on actual spending at NYCERS. You can be sure the Comptroller is not auditing the financial controls at NYCERS. He has enough trouble managing the investment expenses for the pension funds. And, I wouldn’t count on the outside accounting firm either.

My professional opinion is this site is seriously flawed and will never be able to function at any acceptable level as a business disaster recovery site for NYCERS. I suspect that at the time of the lease signing NYCERS had not solved all the operational problems connected to the site and that some of those problems are intractable. Let’s hope NYCERS has the good sense not to extend the lease in 2016.

Annual rents for each year of the ten year lease for 30-00 47th Ave, LIC, NY:

1. $362,076
2. $370,222
3. $378,552
4. $387,070
5. $479,335
6. $490,120
7. $501,148
8. $512,423
9. $523,953
10. $535,742