Showing posts with label LIC. Show all posts
Showing posts with label LIC. Show all posts

Friday, December 9, 2016

Bad News - NYCERS Reup'd on the Lease for the LIC Disaster Site? No. Mazza is Acting Executive Director as of January 1.

In late spring of 2006 NYCERS signed a ten year lease for a disaster recovery site in Long Island City. It came up for renewal this year and it appears from recent comments by the executive director, Diane D'Alessandro that NYCERS renewed the lease.

At the November 10, 2016 Board Meeting, D'Alessandro announced that by January 1, 2017 some poor souls who work in document control and the mail room will be assigned to work at LIC. There was no mention of the Certificate of Occupancy for the site. Maybe NYCERS finally got the C.O. after ten years. Maybe they are afraid to say so because then they would have to admit it took them ten years to get it.

I don't understand how DC-37 has agreed to let their members to be permanently exiled to such a dump. You can be sure none of D'Alessandro's buddies will have to work there. I wounder if the Post Office will pick up mail from the site?

As far I know there has never been a full fledged test of a recovery plan since workers were not legally permitted on the site. I have no idea how NYCERS has been able to safely run their file backup system at this site when there is no permanent staff at the site. As far as I know there is no plan B.

I guess NYCERS had no option but to renew the lease for five years.

Well D'Alessandro will be gone on January 1. It will be someone else's mess then. Oh, maybe Mazza can solve this problem.

Sunday, May 10, 2015

Budget Insanity at NYCERS

On March 12, 2015 the NYCERS Trustees approved the NYCERS FY-2016 administrative budget. It was Diane D'Alessandro's tenth annual adiministrative budget proposal as executive director.

The key numbers are listed in the table below. For comparison purposes I added FY-2006, my last budget as executive director, and FY-1996, NYCERS's last budget under the city's budget structure. The 1996 budget was draconian to say the least. You can see from the table that the NYCERS's budget has increased by 43% over the last ten years or 4.1%/year.

NYCERS: Admin. Budget Over the Years

FY-2016FY-2006FY-1996
FT Positions 401342155
PT Positions 35 130
PS $30.2M $20.0M$6.2M
OTPS $19.4M $14.7M$2.6M
Sub-total $49.6M $34.7M$8.8M
Fringe $8.2M $4.1M$0
Total $57.8M $38.8M$8.8M

What is extraordinary about this budget is that it is the beginning of a delusional spending spree on IT upgrades. It is being called the "Legacy Replacement Project". It proposes to spend $132M over a five year period, staring with $500,000 to be paid to Gartner, Inc. and $1.75M to other consultants to be named later. At the meeting D'Alessandro indicated that she was going to pay $310,000 to an outside firm to write a RFP for this "grand" project. Listed below is the five year spending bonanza.

  • 2016 - $ 2.25M
  • 2017 - $ 9.91M
  • 2018 - $27.86M
  • 2019 - $46.00M
  • 2020 - $46.24M

To OMB's credit they complained about the $310,000 RFP. The trustees, however, approved this budget in spite of making noise about the long term cost of the project.

You would think, considering how well funded that the NYCERS budget is, that the NYCERS staff would be able to write their own RFP. But D'Alessandro got that part right, her staff can't do the job.

Of course, without the results of the RFP where did the $132M amount come from.

In fact over the last ten years D'Alessandro should have been standardizing NYCERS applications on a current client/server platform using a relational database system. This process was underway in 2005 when she first appointed. But thankfully the old CICS/VSAM systems were so well made that they have survived long after the point when they should have been retired. Thank God for small favors.

D'Alessandro, unfortunately, was appointed to her current job in 2005 without any prior relevant experience by Martha Stark and a bunch of compromised trustees. D'Alessandro had never managed anyone. She is not a lawyer, nor an accountant, nor an actuary. She had never worked for a pension system and she had absolutely no IT experience. What she was and still is, is grossly incompetent.

To make the situation worse, the current IT director was working as the NYCERS call center manager when D'Alessandro moved her to the agency's IT division in 2007. At that point she had no IT experience either. Six months later D'Alessandro makes her the director of the IT division. It reminds of when Rudy put a key punch salesman in charge of DoITT. She is as incompetent as D'Alessandro.

D'Alessandro submitted a five page justifying her budget proposal. It is a classic example of using a lot of words to create a fog, lots of hand waving and saying nothing except give me $132M. I actually don't think D'Alessandro wrote the summary. Five pages is far too much work for D'Alessandro. It looks like a cut and paste job from some marketing crap.

This is the justification for $132M.

MAJOR PROJECTS

In a continued effort to provide greater operational efficiency, enhanced customer service and improved quality and production, NYCERS is proposing to undertake a major information technology (IT) and business system replacement project that will streamline and consolidate all applications and functions into a single data repository for common use throughout the agency.

The project which will be known as the Legacy Replacement Project (LRP) will be conducted in three stages over a minimum of five years. The LRP will require a commitment of resources to be phased in over that time.

In order to ensure the success of the LRP, NYCERS is proposing two associated projects including a communications project designed to accommodate expanded electronic interaction with clients and a change management project to address staff training, coaching, and leadership development needs.

You can always spot a IT disaster from the start. It has a long time frame and they don't tell you what they are going to do in the first week of the project, the second week of the project, third week of the project,... You get the picture. Oh yeah, it will also cost a lot of money. I will guarantee you that this project will fail.

When I left NYCERS in 2005, NYCERS had one of the best IT departments in the city. The agency had:

  1. between 60 and 65 IT professionals on staff
  2. an integrated mainframe (VSE) and a network system (Windows Server 2003)
  3. all major work operations automated running under CICS using VSAM and DB2
  4. an advanced web site which allowed members and retirees to view their own specific data.
  5. an imaging/workflow system covering all paper based applications for all members and pensioners.
  6. a dedicated document scanning division
  7. a tested computer disaster recovery plan (we were researching a full business recovery plan and site)
  8. a standardized accounting package running on the network supporting both pension and administrative operations
  9. a nightly backup process of all files including the Exchange Server
  10. a project to upgrade major applications to DB2.
  11. an advanced Avaya based call center with its own backup PBX.
  12. a modern customer service center with its own integrated queuing system
  13. an on-site fully equipped training center.
  14. a proximity card system for both time keeping and security.
  15. a separate network system for the security and a full camera subsystem.

Returning to the executive summary, it is not only short on details about the project but it also has a great deal of deceptive information. The following are a few quotes out of D'Alessandro's write-up along with some clarifying comments.

RECENT HISTORY

Over the past nine years, NYCERS has made significant infrastructure improvements.

Document Management

NYCERS continues to ensure the preservation and security of all vital records and to promote an environmentally sound approach to document management. This effort includes the incorporation of all incoming documents into the NYCEWORK system in an electronic format and the ongoing digitization of historical paper documents.

In 2014, NYCERS prepared, scanned and indexed over 200,000 incoming documents and converted over 1 million historical records into digital images. Since the inception of the document management project, over 38 million paper records have been imaged, saving approximately $600,000 in document storage fees.

From this quote you would think that document management was introduced during the last few years. In fact, it was there when D'Alessandro walked in the door. The system in place at that time used a software package called Staffware. It was working well but it was being stressed by the volume of documents that NYCERS was dealing with. It actually continued to function up until 2014 when a new software package, Filenet/NYCEWORK, was finally put into production. I wonder where the $600,000 in saved storage fees came from?

Unfortunately, while Staffware needed to be either upgraded or replaced, Filenet runs significantly slower than Staffware did. That in turn has created incremental delays at every step of every process in the agency. It has created havoc with the production levels for all major functions in the agency.

This is an example of a mismanaged project being put into production so that management doesn't have to say it made a mistake. We saw this problem with the CityTime project.

While we are talking about mistakes, there have been two major IT disasters at NYCERS in the last nine years costing millions of dollars. One was the DB2 project and the other is the infamous Long Island City disaster recovery site. At least when the DB2 project was cancelled, the bleeding stopped. The LIC-DR project is a never ending cancer.

When I left NYCERS in 2005, the IT department with the help of two consultants was slowly converting over our major VSAM files to a DB2 relational database platform. We had already had two significant systems running under DB2 but there was a long term need to shift all major files over to the relational model. Two months after I left the project was cancelled. Subsequently NYCERS entered into a contract with IBM/Blue-Phoenix to do the DB2 conversion. That project dragged on for years, cost millions, and eventually failed. NYCERS even tried to sue IBM but with very little relief.

Business Continuity

The Long Island City business recovery site enables NYCERS to restore business operations and critical applications in the event of a disaster. This ensures the sustainability of operations and reduces the potential for data loss. Business recovery time is within four hours and the data recovery point is within 30 minutes.

From this quote you would conclude LIC-DR is a success. That is not true in any sense.

The lease on the LIC-DR project was signed in 2006 and NYCERS still does not have a Certificate of Occupancy for the site. I have previously written about this mess. It is not clear what is the legal status of having employees on that site.

For the record, the rent at LIC-DR is $530,000 for FY-2016 plus another $400,000 for other ancillary lease charges. The site has serious water leakage, asbestos, and plumbing problem. The lease comes up for renewal in the spring of 2016. Let's see what the decision will be then.

And in this season of delusion, NYCERS is planning a tertiary disaster recovery site. They even started buying hardware ($250,000 in FY-2015) for this mythical site. Some people just don't get it.

Business Process Automation

NYCERS continues to streamline business processes and leverage infrastructure investments to improve service delivery. As a result, NYCERS has achieved efficiencies in areas including loans and survivor benefits. On-line loan processing time has been reduced from 10 business days to five business days.

In this quote D'Alessandro appears to claim to have automated loans and survivor benefits in the last nine years and that the turnaround time on loans has been dropped from 10 to 5 days. Actually in 2005 both loans and survivor benefits were fully automated. The loans were the first applications automated in the early 1980's and also the first application to be integrated into the imaging/workflow system in 2002. In 2005, if you filed a loan application by Tuesday, NYCERS mailed a loan check to you that Friday. You would wonder why D'Alessandro would have made the above statement.

While all this craziness was going on with the great IT upgrade, D'Alessandro also threw in a request for 12 new employees with salaries of $720,000. No explanation was given. This was on top of 6 new employees in FY-2015, again with no explanation. Since 2013 the full time staff has gone from 372 to 401 and part time staff from 12 to 35. Why are there back logs with production work? Can you imagine what A.C.S. could do with that kind personnel increase.

Thursday, September 5, 2013

The Haze Over the Disaster Recovery Site at Long Island City - 2013

On March 14, 2013 the NYCERs Board of Trustees adopted resolutions approving the NYCERS administrative and loan operating budgets for FY-2014. The administrative budget was $44.6M with an added $7.6M for fringe benefits. The loan budget was $1.7M with $.3M for fringe benefits. The loan budget is totally paid by the active members through fees for loans. The executive director, Diane D’Alessandro, provided a two and a half page justification for these expenditures.

2011

In September 12, 2011 posting, I wrote about the fiasco surrounding the NYCERS disaster recovery (DR) site at Long Island City (LIC). At the time there was a five year delay in opening the DR site.

2012

In her February, 2012 justification for the FY-2013 NYCERS administrative budget D’Alessandro referred to the DR site issue as follows:

“The build-out of the business continuity site is nearing completion. Once the construction phase is finalized, the installation of the secondary data center will commence. This final phase will extend through FY-2103. Upon completion, the secondary data center will be equipped with a mainframe computer and have the capability for site-to-site data replication and data updating allowing for the eventual elimination of the Sterling Forest backup site. In an emergency, the LIC will have the capacity to house approximately 300 staff working on a two shift model. During normal business, the facility will be utilized for document management, training and IT testing and backup functions.“

In a July 13, 2012 posting, I commented on D’Alessandro’s failure to address the then six year delay in opening the LIC DR site and all the other problems surrounding this project.

2013

Part of the FY-2014 administrative budget is a $2.9M non payroll (OTPS) cost for maintaining the LIC DR site. In her FY-2014 budget justification D’Alessandro makes only the following mention of the LIC site:

“With the completion of the secondary data center at Long Island City (LIC), NYCERS will shift its focus to restructuring the Adams Street data center. Proposed upgrades will improve the utilization of space and enhance cooling, heating and air flow systems. This effort will provide greater energy efficiency, expanded storage capacity and improved functionality.”

From this terse comment you might assume that the DR site is fully functional. You will notice, however, that there is no mention of the actual completion date of the site, the award of a certificate of occupancy or the passage of fire inspection for the LIC site. There is also no mention of maximum occupancy for the site and there is no mention of a full operational test of the site. Remember the original 2006 lease renewal comes up in FY-2016, just two years away.

Given the history of this project it is quite possible that the site is not operational at all in spite of D’Alessandro’s attempt to give that impression. Of course it now seems that replacing the air conditioning unit in the computer room at Adams Street is the new hot item. At least, it appears that is what she is talking about. "Restructuring" and "Upgrades" are not words loaded with a lot of information.

Friday, July 13, 2012

Background Update - Disaster Recovery Site: Has NYCERS Fixed the Problem?

Update - July 24, 2018

While doing some research on the current "Legacy Replacement" fiasco, I came across an item in the FY-2012 NYCERS financial statement (CAFR). It was for the cost of the build out for the Long Island City (LIC) disaster recovery site. That cost was $3,791,813 paid out as of June 30, 2012. You will never hear any NYCERS staff mention this number in conjunction with LIC.

End of update.

Original post - July 13. 2012

Last week I received a copy of the NYCERS Administrative Budget for FY-2013.

In September, 2011, I commented on the five year delay in completing the NYCERS Long Island City (LIC) disaster recovery (DR) site.

Based on the NYCERS budget proposal for 2012-2013, the site is now “nearing” completion with a date sometime before June 30, 2013.

There continues to be an ongoing confusion about the completion of this project. On March 25, 2012 in response to a newspaper article about the five year delay in getting the LIC site operational Karen Mazza, the NYCERS legal director, stated that the site would be ready to open on April 1, 2012. Considering the history of this project we’ll need to see the certificate of occupancy before being sure that the site is open. Of course, having a fully functioning alternative site for all agency operations is a whole other story.

The rent for the LIC site for the coming year is $496,500.

In its FY-2013 budget NYCERS’s has increased its headcount to a total of 394 F/T and 25 P/T positions. In addition there are always a significant number of consultants working at NYCERS.

As per the budget document the DR site is projected to only support 150 people at any one time or 300 people using a two shift schedule. The certificate of occupancy and the FDNY inspection will actually determine the working capacity of the site.

The two shift schedule raises contract issues with the unions involved and creates serious hardships for employees stuck with off hours schedules. It also leaves a problem for the remaining 94 employees and consultants and their associated work.

Part of the long delayed plan is to install a second mainframe at LIC with the idea of ending the use of the IBM DR site at Sterling Forest, NY.

This is fundamentally a bad idea. The elimination of the Sterling Forest site means that NYCERS will lose flexibility in its ability to respond to unknown disasters. It also creates personnel, management, and cost challenges in keeping the LIC mainframe in synch with the Adams Street machine. NYCERS is a relatively small organization which needs to avoid high overhead and to leverage its skills with the use of the Sterling Forest site.

Prior to 2005 NYCERS had an IT expert as the executive director. The current executive director has no business experience prior to 2005. This makes NYCERS very susceptible to “pie in the sky” schemes as evidenced by the six year delay in getting this project off the ground.

As per the budget document, NYCERS is also planning to move its document management, training, and IT testing/backup functions to this site.

It is not clear what NYCERS will do with its modern training center at Adams Street. Adams Street is the site where all NYCERS employees are currently working. It is clearly a more efficient site to train employees than LIC.

Moving document management to Long Island City will only create delays in agency functions and increase the difficulty in controlling documents that arrive at Adams Street and then will have to be transported to LIC site for processing.

The current offsite record storage at Bergen Street will be moved to the DR site. This was a pre-2005 concept. I wonder how it survived.