Deduplication is on primary storage now!!!

Deduplication is the technology that singles out the data that needs to be backed up. The ‘duplicate’ data is identified by patterns or by generating a unique hash for each data set and verifying against the existing data sets. Usually Deduplication software is bundled with VTLs, D2Ds which are nothing but secondary storage.

NetApp has come up with a Deduplication solution (A-SIS) that operates not only on secondary storage but also on primary storage. NetApp has integrated the solution with Data ONTAP® software and the WAFL® file system. Have a look at the following NetApp webpage to know more about the product from NetApp.

http://www.netapp.com/products/storage-systems/near-line-storage/asis-dedup.html

Polysius is one of the leading engineering companies equipping the cement and minerals industries. Polysius has tried out the solution. To know more about its experience and an overview of NetApp’s solution visit the following link.

http://www.enterprisestorageforum.com/sans/features/article.php/3707306

What can be cooked?

Deduplication is always an interesting technology which is getting quickly adopted. When data growth is in exponential, Deduplication is one among the solution to put a check. It started all the way on operating at secondary storage and now into primary storage.

I think this intelligence piece can be inserted into Operating Systems, Databases and applications that handle internal storage architecture like Exchange. These inventions will really shrink the data to be stored and can check the growth of Storage.

Lets wait and watch where we going!!!

Deduplication is on the rise

Data center professionals are struggling to keep the growth of Secondary Storage in control. The volume of data being generated is growing exponentially and needs a good methodology to check the explosion. Governments add more woe to this situation in terms of regulatory compliance. For example Banks has to keep all the details of a users for 10 years.

 

By taking a closer look on what is being backed up, there are chances of backing up the duplicate data, data that doesn’t change between backup times, data that is changed, new data added in the repository, etc. Data center managers often use tape to backup the data, which has its own advantages and disadvantages. The main issue is the time for restoration of lost data which gives rise to VTLs and Disk to Disk backup (D2D). Either case duplicate or data that doesn’t touched for a time period will get backed up.

 

Deduplication

To overcome this situation we need a technique which can store more data by analyzing the unique repetitive patterns which is nothing but deduplication. Deduplication is nothing but a software application that singles out the data that needs to be backed up. There are quite few more names like single instance storage, Record Linkage. Commonality Factoring, etc. Deduplication can be deployed in Disk to Disk Backup (D2D) environment or VTLs that emulate the tape libraries.

 

There are two categories of deduplication technology is available. One is to match the patterns at the byte level and the other is to generate a hash for a incoming data set. Both the technologies has got its one advantages and disadvantages. Choosing one from the category should be made by the requirements and performance needed.

 

Fears

There are quite a few fears over deduplication. Data integrity, there is always only one copy of data, what happens if the deduplication application fails, may increase the backup window, etc.

 

The market segment is alive and moves quickly towards maturization. Biggies are coming up with their products that contains deduplication. To name a few, HyperFactor from Diligent and Axion from Diligent Avamar(Acquired by EMC).

FCoE Catches UP

FCoE

FCoE stands for Fibre Channel over Ethernet. The idea is to leverage Ethernet (only TCP) for transferring FC Packets. The assumption is that FCoE will run on 10 Gig Ethernet. The project has been proposed in T11 standard body. The FCoE standard is proposed in April 2007 and FCoE products are expected by 2009. The ratified standard may be available by fourth quarter of 2008. Currently there are no products available that supports FCoE in market.

FCoE gets a place inside the IPSAN along with iSCSI, iFCP and FCIP. But FCoE has got its own advantages and disadvantages.

Advantages:

1. The existing Ethernet can be leveraged.

2. There is mapping between Ethernet Frame and FC frame which means FCoE will be simpler and faster.

3. Stripes the overhead of IP.

Disadvantages:

1. Inside Ethernet, data loss cant be predicted. But SAN can never accommodate any loss in data. But this can be averted by adopting “Pause Frames” mechanism.

2. The assumption on FCoE is to leverage 10 Gig Ethernet which may be off enterprise level.

    

On the industries front, they are catching up FCoE. There was a demonstration by Qlogic, NetApp and Nuova Systems in SNW using FCoE adapters. Biggies like IBM, EMC, Sun Microsystems, Intel are backing up the idea.

Visit www.fcoe.com to learn more.

SNW

Storage Networking World

SNW, an event that spans about a week, where the storage professionals meet, share and learn. Various companies will be showcasing their solutions and products in this event. There is a possibility to attend various education sessions (over 140), learn the capabilties of various companies, meet peers.

Visit www.snwusa.com to learn more and to register for participating. Also visit http://www.snwusa.com/agenda.html to know more about the agenda.

 

Whats cooking in SNW?

Here is a small list of expos.

1. Brocade showcases 8 Gbps FC Blades for its 48000 Director. (http://www.brocade.com/products/directors/silkworm_48000/index.jsp)

    

2. For the first time, SNIA will demonstrate the capabilities of XAM specification using the applications developed by EMC, HP, SUN and Vignette.

    

3. Hitachi unveils its Simple Modular Storage Model 100 which can support four servers and protect up to 9TB of data.

   (http://www.hds.com/products/storage-systems/simple-modular-storage.html)

    

4. EMC makes NetWorker backup application stronger by adding data technologies that was aquired during 2006.

 

SNIA’s Journey Continues…

SNIA’s Decade

SNIA is a non commercial industry organization formed in 1997. A small group of industry leaders and companies came together to form a consortium to advance the storage industry. Currently SNIA has a breadth of 400 member companies and more than 7000 individuals under its wing. The mission of the SNIA is to develop various standards and technologies to promote and unite the storage industry.

SNIA has made a remarkable journey to bring up and unite the storage industry in the last ten years. Here are few things to share from my knowledge.

Mile Stones

Here is a list of few major mile stones that are reached in this decade by SNIA.

1. The development of Storage Management Initiative Specification (SMI-S).

2. Accredition of SMI-S from International Standards Organization (ISO) and International Electrotechnical Commission (IEC)

3. Certifying more than 10000 Professionals and 300 Products

4. Launching of Standard Storage Management Framework

5. Development of eXtensible Access Method (XAM) API Specification

6. Creating the regional affiliates in Europe, India, China, Japan, South Asia, Australia and few more countries.

7. Common RAID Disk Data Format Specification

 

Key Active areas

SNIA will concentrate on Management Framework, XAM apart from its regular activities on SMI-S. There are lot of key initiatives are coming up and few note worthies are;

1. Green Storage Initiative (http://www.snia.org/forums/green/)

2. Data – Deduplication and Space Reduction Special Interest Group. (http://www.snia-dmf.org/dpi/DDSR-SIG.shtml)

3. Storage Security forum (http://www.snia.org/forums/ssif/)

To know more about SNIA and its activities, visit www.snia.org