Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000420FlexRAID RAID-F[All Projects] Generalpublic2016-03-14 17:362017-11-23 14:01
Reportertonym 
Assigned ToBrahim 
PriorityurgentSeveritymajorReproducibilitysometimes
StatusresolvedResolutionopen 
PlatformHP MicroServerOSWindowsOS VersionWin7 Home x64
Summary0000420: Update corrupts PPU for previous file(s) processed during Update
DescriptionEver since migrating from FlexRAID-2.0 to FlexRAID-2.1-Final-2015-11-01, I have been plagued by unexpected Validate and (and thus also Verify) Errors. After many many frustrating months, numerous Re-Creates, and controlled experiments, I have been able to isolate the problem to Validate errors with some files added to the RAID during the previous Update.
Steps To ReproduceA typical scenario to reproduce the problem would be:
1. Re-Create the RAID, followed immediately by full Verify. Everything is fine.
2. Add a new sub-directory e.g. L:\Data1\Newdir1\, with say a dozen files total
   around 4GB, all to the same physical disk.
3. Run Update followed by Validate, and everything is fine.
4. Add a new sub-directory, but this time to a different disk
   e.g. M:\Data2\Newdir2, again say half a dozen files around 4GB.
5. Run Update followed by Validate, and then FlexRAID flags one or two of the
   files in L:\Data1\Newdir1\ as corrupt, i.e. files added before the previous
   Update, and which had already checked out OK by the previous Validate.
Note that none of these original files are actually corrupt; they check out using independent PAR2 files.
Additional InformationI am running RAID-F and Storage Pooling in Expert mode on a dedicated Windows 7 Home x64 based system. I have turned off my AV software, turned off Windows Update, so there should be nothing left to interfere with my RAID disks. I am 100% confident that my disks are all good (S.M.A.R.T. tests etc.), and that none of my files are corrupt; I can check any that FlexRAID flags as corrupt using my own independent PAR2 files.

My workaround has been to revert to FlexRAID-2.0-Final_2014-08-16, which has now been running faultlessly for about a month.
I've been using FlexRAID from 2011, since before it became commercial; I only started experiencing these problems with v2.1.

I have a suspicion that this second batch of files doesn't need to be to a different disk, just different sub-directory e.g. L:\Data1\Newdir2; but I no longer have a FlexRAID-2.1 system to test this hypothesis against.
TagsNo tags attached.
RAID-F Version 2.1
Attached Files

- Relationships

-  Notes
(0001620)
Brahim (manager)
2017-01-07 08:14
edited on: 2017-01-07 08:27

Thanks for the report.
Your test scenario will be executed within the next few days.
We'll update once done.

Do you happen to have the logs of your run in TRACE mode?
This would be very useful if provided.

(0001738)
dEadY (reporter)
2017-07-22 20:30
edited on: 2017-07-22 21:41

I just noticed the same issue with my installation. I very rarely use my PC, and for a very long time I haven't used "Validate" or "Verify". The last time was:
[2015-10-04 23:23:42,938] INFO : [validate] started at: Sun Oct 04 23:23:42 CEST 2015
And besides some "NEW" files listed in the logs (bad date, I'm sure), no corruption has been detected.

I haven't run those commands at all since I upgraded to FlexRAID-2.1-Final-2015-11-01 on 2016-07-03:
[2016-07-03 14:47:43,337] INFO : FlexRAID 2.0 final [2014-08-16] [Snapshot 1.4 stable / Storage Pool 1.0 stable / Real-Time 1.0 experimental]
[2016-07-03 14:54:56,105] INFO : FlexRAID 2.1 [Snapshot 1.1 / Storage Pool 2.0] [2015.11.01]

Today I've run Verify, and after an hour already it tells me about bit mismatches. If I run Validate, I get hundreds of lines in the logs that look like this one:
[2017-07-22 18:10:06,974] ERROR: Corrupted: C:\FR\D1\VirtualMachines\Windows-x64\Win-x64-s113.vmdk

In the set of corrupted files, there are some bigger 7-zip and RAR archives, which validate as fully correct. For some backup images, which have a checksum and can also be verified for correctness using the backup tool, I also get 0 bit errors, although more than 27 of the 36 files of one single backup are flagged as corrupt in the logs (this backup image was created just a few months ago).
As another example, one Windows installation ISO is affected, and I verified the SHA1 hash with a public source, and it matches:
[2017-07-22 14:16:21,658] ERROR: Corrupted: C:\FR\D2\E-Downloads\web-downloads\X17-59885.iso --> SHA1: 39D2E2924E186124EA44D2453069B34EF18EA45E
The "last changed timestamp" of the folder in which the ISO is located is dated 1y before my FlexRaid update to 2.1 happened.

In addition, a lot of (small) files from one specific folder have been marked as corrupted. Interestingly, looks like we're talking about a file from a folder that (judging from old logs) has been moved at some point:
[2016-05-01 06:25:14,247] WARN : C:\FR\D4\movies\_backup\Nexus5\WhatsApp\Media\WhatsApp Images\IMG-20150214-WA0037.jpg no longer exist! Skipping...
[2017-07-22 17:20:12,038] ERROR: Corrupted: C:\FR\D4\Backup\Nexus5\WhatsApp\Media\WhatsApp Images\IMG-20150214-WA0037.jpg

So looks like it's not related to file size, and as files from 2014 are also affected, I'm not sure whether this issue can be ruled out for previous releases, but it looks at least that a big amout of the affected files have been created after I updated to FlexRaid 2.1.

I kept all logs since my installation in October 2014 (Expert Mode), so I can grep them if needed. I'm also quite sure I never ran a rebuild of the PPU or forced sync verification (but I can search in the logs if you tell me what to look for).

Can I help with your investigation somehow? I'd like to rebuild the PPU quite soon and afterwards add a 6TB disk to the array. Would you suggest me to downgrade the FlexRaid version?

My configuration is 5x3TB DRUs und 1x3TB PPU on an upgraded Windows 10 x64 machine, no S.M.A.R.T. errors for any of the disks (they're all equal, TOSHIBA DT01ACA300).

Edit: The "NEW" files were already present, removed the section here.

(0001739)
tonym (reporter)
2017-11-22 17:56

Does the Refresher Release 2017-11-21 fix this problem? Nothing in the change log to indicate that it does.
(0001742)
Brahim (manager)
2017-11-23 14:01

Fixed in build 2017-11-21

- Issue History
Date Modified Username Field Change
2016-03-14 17:36 tonym New Issue
2016-03-16 07:51 tonym Issue Monitored: tonym
2016-03-16 07:51 tonym Issue End Monitor: tonym
2017-01-07 08:14 Brahim Note Added: 0001620
2017-01-07 08:14 Brahim Assigned To => Brahim
2017-01-07 08:14 Brahim Status new => feedback
2017-01-07 08:27 Brahim Note Edited: 0001620 View Revisions
2017-07-22 20:30 dEadY Note Added: 0001738
2017-07-22 20:39 dEadY Note Edited: 0001738 View Revisions
2017-07-22 20:41 dEadY Note Edited: 0001738 View Revisions
2017-07-22 20:42 dEadY Note Edited: 0001738 View Revisions
2017-07-22 20:48 dEadY Note Edited: 0001738 View Revisions
2017-07-22 20:51 dEadY Note Edited: 0001738 View Revisions
2017-07-22 20:52 dEadY Note Edited: 0001738 View Revisions
2017-07-22 21:05 dEadY Note Edited: 0001738 View Revisions
2017-07-22 21:18 dEadY Note Edited: 0001738 View Revisions
2017-07-22 21:19 dEadY Note Edited: 0001738 View Revisions
2017-07-22 21:19 dEadY Note Edited: 0001738 View Revisions
2017-07-22 21:41 dEadY Note Edited: 0001738 View Revisions
2017-11-22 17:56 tonym Note Added: 0001739
2017-11-22 17:56 tonym Status feedback => assigned
2017-11-23 14:01 Brahim Note Added: 0001742
2017-11-23 14:01 Brahim Status assigned => resolved


Copyright © 2000 - 2010 MantisBT Group
Powered by Mantis Bugtracker