|Anonymous | Login | Signup for a new account||2017-08-21 19:29 UTC|
|Main | My View | View Issues | Change Log | Roadmap|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0000420||FlexRAID RAID-F||[All Projects] General||public||2016-03-14 17:36||2017-07-22 20:30|
|Platform||HP MicroServer||OS||Windows||OS Version||Win7 Home x64|
|Summary||0000420: Update corrupts PPU for previous file(s) processed during Update|
|Description||Ever since migrating from FlexRAID-2.0 to FlexRAID-2.1-Final-2015-11-01, I have been plagued by unexpected Validate and (and thus also Verify) Errors. After many many frustrating months, numerous Re-Creates, and controlled experiments, I have been able to isolate the problem to Validate errors with some files added to the RAID during the previous Update.|
|Steps To Reproduce||A typical scenario to reproduce the problem would be:|
1. Re-Create the RAID, followed immediately by full Verify. Everything is fine.
2. Add a new sub-directory e.g. L:\Data1\Newdir1\, with say a dozen files total
around 4GB, all to the same physical disk.
3. Run Update followed by Validate, and everything is fine.
4. Add a new sub-directory, but this time to a different disk
e.g. M:\Data2\Newdir2, again say half a dozen files around 4GB.
5. Run Update followed by Validate, and then FlexRAID flags one or two of the
files in L:\Data1\Newdir1\ as corrupt, i.e. files added before the previous
Update, and which had already checked out OK by the previous Validate.
Note that none of these original files are actually corrupt; they check out using independent PAR2 files.
|Additional Information||I am running RAID-F and Storage Pooling in Expert mode on a dedicated Windows 7 Home x64 based system. I have turned off my AV software, turned off Windows Update, so there should be nothing left to interfere with my RAID disks. I am 100% confident that my disks are all good (S.M.A.R.T. tests etc.), and that none of my files are corrupt; I can check any that FlexRAID flags as corrupt using my own independent PAR2 files.|
My workaround has been to revert to FlexRAID-2.0-Final_2014-08-16, which has now been running faultlessly for about a month.
I've been using FlexRAID from 2011, since before it became commercial; I only started experiencing these problems with v2.1.
I have a suspicion that this second batch of files doesn't need to be to a different disk, just different sub-directory e.g. L:\Data1\Newdir2; but I no longer have a FlexRAID-2.1 system to test this hypothesis against.
|Tags||No tags attached.|
edited on: 2017-01-07 08:27
Thanks for the report.
Your test scenario will be executed within the next few days.
We'll update once done.
Do you happen to have the logs of your run in TRACE mode?
This would be very useful if provided.
edited on: 2017-07-22 21:41
I just noticed the same issue with my installation. I very rarely use my PC, and for a very long time I haven't used "Validate" or "Verify". The last time was:
[2015-10-04 23:23:42,938] INFO : [validate] started at: Sun Oct 04 23:23:42 CEST 2015
And besides some "NEW" files listed in the logs (bad date, I'm sure), no corruption has been detected.
I haven't run those commands at all since I upgraded to FlexRAID-2.1-Final-2015-11-01 on 2016-07-03:
[2016-07-03 14:47:43,337] INFO : FlexRAID 2.0 final [2014-08-16] [Snapshot 1.4 stable / Storage Pool 1.0 stable / Real-Time 1.0 experimental]
[2016-07-03 14:54:56,105] INFO : FlexRAID 2.1 [Snapshot 1.1 / Storage Pool 2.0] [2015.11.01]
Today I've run Verify, and after an hour already it tells me about bit mismatches. If I run Validate, I get hundreds of lines in the logs that look like this one:
[2017-07-22 18:10:06,974] ERROR: Corrupted: C:\FR\D1\VirtualMachines\Windows-x64\Win-x64-s113.vmdk
In the set of corrupted files, there are some bigger 7-zip and RAR archives, which validate as fully correct. For some backup images, which have a checksum and can also be verified for correctness using the backup tool, I also get 0 bit errors, although more than 27 of the 36 files of one single backup are flagged as corrupt in the logs (this backup image was created just a few months ago).
As another example, one Windows installation ISO is affected, and I verified the SHA1 hash with a public source, and it matches:
[2017-07-22 14:16:21,658] ERROR: Corrupted: C:\FR\D2\E-Downloads\web-downloads\X17-59885.iso --> SHA1: 39D2E2924E186124EA44D2453069B34EF18EA45E
The "last changed timestamp" of the folder in which the ISO is located is dated 1y before my FlexRaid update to 2.1 happened.
In addition, a lot of (small) files from one specific folder have been marked as corrupted. Interestingly, looks like we're talking about a file from a folder that (judging from old logs) has been moved at some point:
[2016-05-01 06:25:14,247] WARN : C:\FR\D4\movies\_backup\Nexus5\WhatsApp\Media\WhatsApp Images\IMG-20150214-WA0037.jpg no longer exist! Skipping...
[2017-07-22 17:20:12,038] ERROR: Corrupted: C:\FR\D4\Backup\Nexus5\WhatsApp\Media\WhatsApp Images\IMG-20150214-WA0037.jpg
So looks like it's not related to file size, and as files from 2014 are also affected, I'm not sure whether this issue can be ruled out for previous releases, but it looks at least that a big amout of the affected files have been created after I updated to FlexRaid 2.1.
I kept all logs since my installation in October 2014 (Expert Mode), so I can grep them if needed. I'm also quite sure I never ran a rebuild of the PPU or forced sync verification (but I can search in the logs if you tell me what to look for).
Can I help with your investigation somehow? I'd like to rebuild the PPU quite soon and afterwards add a 6TB disk to the array. Would you suggest me to downgrade the FlexRaid version?
My configuration is 5x3TB DRUs und 1x3TB PPU on an upgraded Windows 10 x64 machine, no S.M.A.R.T. errors for any of the disks (they're all equal, TOSHIBA DT01ACA300).
Edit: The "NEW" files were already present, removed the section here.
|2016-03-14 17:36||tonym||New Issue|
|2016-03-16 07:51||tonym||Issue Monitored: tonym|
|2016-03-16 07:51||tonym||Issue End Monitor: tonym|
|2017-01-07 08:14||Brahim||Note Added: 0001620|
|2017-01-07 08:14||Brahim||Assigned To||=> Brahim|
|2017-01-07 08:14||Brahim||Status||new => feedback|
|2017-01-07 08:27||Brahim||Note Edited: 0001620||View Revisions|
|2017-07-22 20:30||dEadY||Note Added: 0001738|
|2017-07-22 20:39||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 20:41||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 20:42||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 20:48||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 20:51||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 20:52||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 21:05||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 21:18||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 21:19||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 21:19||dEadY||Note Edited: 0001738||View Revisions|
|2017-07-22 21:41||dEadY||Note Edited: 0001738||View Revisions|
|Copyright © 2000 - 2010 MantisBT Group|