Skip to content

Xfs support #1476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 39 commits into from
Closed

Xfs support #1476

wants to merge 39 commits into from

Conversation

isciurus
Copy link

Implemented and tested: blkstat, blkls, blkcat, istat, ils, icat, fls, fsstat.

Tested on 6 images created from CentOS, including one with ~111000 files and nested directories to test multilevel B+trees and various XFS corner cases (large btree inodes and directory blocks).

It is in a good shape to build timeline with mactime/fls from a raw image during investigations.

Unsupported XFS features (yet): real-time devices, file attributes parsing.

Implemented: blkstat, blkls, blkcat, istat, ils, icat, fls, fsstat.

Unsupported XFS features (yet): very large inodes (XFS_DINODE_FMT_BTREE), real-time devices, journaling log
Implemented: blkstat, blkls, blkcat, istat, ils, icat, fls, fsstat.

Unsupported XFS features (yet): very large inodes (XFS_DINODE_FMT_BTREE), real-time devices, journaling log
- Fixed a few int overflows
- Adjusted addressing scheme for cases of AG size not being power of 2
- Added proper last block and last inode calculationwq
- Now parsing all XFS_DINODE_FMT_EXTENTS forms of directories properly ("block directories", "leaf directories", "node directories"). Would benefit from refactoring
- Finished XFS_DINODE_FMT_BTREE directory blocks parsing
- A bit of refactoring
@isciurus
Copy link
Author

I realize now there is another pull request, where we work on the same filesystem independently (#1461). I look forward to collaborate, looks like I haven't tested tsk_recover functionality, but large directories and timelining work in my case.

@isciurus isciurus force-pushed the xfs_support branch 3 times, most recently from 006ab56 to 5828454 Compare April 4, 2019 17:26
@dndusdndus12 dndusdndus12 mentioned this pull request Apr 7, 2019
@kouril
Copy link

kouril commented Apr 25, 2019

I've tested the code and find it quite usable for XFS analysis. I've ran into one problem however (related to handling of unused data in directory records, when the offset is wrongly computed). The following patch fixes the behavior:

--- a/tsk/fs/xfs.cpp
+++ b/tsk/fs/xfs.cpp
@@ -2719,7 +2719,7 @@ xfs_dir_open_meta(TSK_FS_INFO * a_fs, TSK_FS_DIR ** a_fs_dir,

                     if (tsk_verbose) { tsk_fprintf(stderr, "offset_in_block = % is a free space, shifting forward by tsk_getu32(TSK_BIG_ENDIAN, &data_unused->length)) = %d \n", offset_in_block, tsk_getu32(TSK_BIG_ENDIAN, &data_unused->length)); }
                     offset_in_block +=
-                        tsk_getu32(TSK_BIG_ENDIAN, &data_unused->length);
+                        tsk_getu16(TSK_BIG_ENDIAN, &data_unused->length);
                 }
                 else
                 {

@isciurus
Copy link
Author

isciurus commented Sep 9, 2019

Added and tested support for V5 (RHEL), real-time devices (some nuances here), and file attributes

@halpomeranz
Copy link

I've noted two issues where istat seems to not display blocks correctly.

  1. Large file (about 10MB), no direct blocks displayed:
# istat /dev/mapper/vg_root-lv_root 33899988
Inode: 33899988
Allocated
uid / gid: 0 / 0
mode: rrw-r--r--
Flags:
size: 10607741
num of links: 1

Inode Times:
Accessed:       2020-05-31 21:49:48.563830657 (UTC)
File Modified:  2020-05-10 02:16:09.215686581 (UTC)
Inode Modified: 2020-05-29 14:26:16.125125746 (UTC)
File Created:   2020-04-26 04:45:19.727651716 (UTC)

Direct Blocks:

The issue here seems to be that the file is so fragmented, the extents are being stored as a B-Tree. Here's the relevant output from xfs_db:

# xfs_db -r /dev/mapper/vg_root-lv_root
xfs_db> inode 33899988
xfs_db> print
[...]
v3.inumber = 33899988
v3.uuid = 66cc02db-6ab9-49ef-a63e-4f991bf47189
u3.bmbt.level = 1
u3.bmbt.numrecs = 1
u3.bmbt.keys[1] = [startoff] 1:[0]
u3.bmbt.ptrs[1] = 1:4575812
  1. In at least some cases istat is displaying erroneous block numbers:
# istat /dev/mapper/vg_root-lv_root 34569671
Inode: 34569671
Allocated
uid / gid: 0 / 0
mode: rrw-------
Flags:
size: 10733
num of links: 1

Inode Times:
Accessed:       2020-06-03 23:07:59.027530093 (UTC)
File Modified:  2020-06-03 23:07:59.027530093 (UTC)
Inode Modified: 2020-06-03 23:07:59.027530093 (UTC)
File Created:   2019-03-13 16:16:07.403989400 (UTC)

Direct Blocks:
3534879 3559332 3565012

xfs_db disagrees:

xfs_db> inode 34569671
xfs_db> print
[...]
v3.inumber = 34569671
v3.uuid = 66cc02db-6ab9-49ef-a63e-4f991bf47189
u3.bmx[0-2] = [startoff,startblock,blockcount,extentflag] 0:[0,4321567,1,0] 1:[1,4346020,1,0] 2:[2,4351700,1,0]

I've done the math and what istat is displaying are not physical block offsets either. I have no idea what those numbers are.

Is it just that istat is having trouble dealing with fragmentation in general?

@halpomeranz
Copy link

blkls fails to gather all of unallocated:

# df -h /mnt/test
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      499M   75M  425M  15% /mnt/test
# file -s /dev/loop1
/dev/loop1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
# blkls /dev/loop1 >blkls-test
# ls -lh blkls-test
-rw-r--r-- 1 root root 243M Jun 12 20:47 blkls-test

The size of the output file should equal the amount available as reported by the df command. The blockls output is only slightly over half of the available space in the file system.

@halpomeranz
Copy link

This one is more a feature request. It would be great if fls could parse and display deleted directory entries where possible.

@kouril
Copy link

kouril commented Jun 15, 2020

I've noted two issues where istat seems to not display blocks correctly.

from my quite limited experience with hacking the XFS support, I'd say that the particular dataset where you're observing the issues would help identify the problem in the code. Any chance you could share the image?

@halpomeranz
Copy link

I've noted two issues where istat seems to not display blocks correctly.

from my quite limited experience with hacking the XFS support, I'd say that the particular dataset where you're observing the issues would help identify the problem in the code. Any chance you could share the image?

I wish I could. Unfortunately it's customer data in a current investigation.

The system is SLES 12.3. The issues I'm finding seem to be with fragmented files, but that could just be coincidence.

@isciurus
Copy link
Author

@halpomeranz thank you for a thorough review. Is there any chance you can send at least some verbose output -v from istat / blkls along with fsstat info?

@simsong
Copy link
Member

simsong commented Sep 8, 2024

Also, how his this different from #1461?

@simsong simsong mentioned this pull request Sep 8, 2024
@simsong
Copy link
Member

simsong commented Sep 8, 2024

Hi. I'm sorry for the delay in getting to this. If you can give us a test image and a test case and rebuase this to be the current develop release, I can accept it.

@simsong simsong marked this pull request as draft September 8, 2024 18:00
@simsong
Copy link
Member

simsong commented Jan 19, 2025

Closing as we went with #1461.

@simsong simsong closed this Jan 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants