1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
|
# SQLiteFS Implementation Notes
**IMPORTANT**: Always remove completed items from this file. Completed work should not clutter the active notes.
## ZEROBLOB Issue with C String Handling
**Problem**: When using SQLite's `ZEROBLOB(n)` function to extend files with null bytes, the resulting data gets truncated at the first `\0` byte when read through the C API.
**Root Cause**: The SQLite C API treats blob data as null-terminated strings in some contexts. When our `readFileContentAtOffset` function calls `row.get([]const u8, 0)`, it stops reading at the first null byte, effectively truncating the extended portion of the file.
**Current Behavior**:
- File truncation to smaller sizes works correctly (uses `SUBSTR`)
- File extension with `ZEROBLOB` creates the correct data in the database
- Reading the extended file returns only the content up to the first null byte
**Potential Solutions**:
1. Use SQLite's blob-specific API functions that handle binary data with explicit length
2. Replace ZEROBLOB with a different padding strategy (e.g., spaces, repeated single chars)
3. Use SQLite's `sqlite3_column_bytes()` to get actual blob length instead of relying on null termination
4. Implement proper binary blob handling in the zqlite wrapper
**Impact**: File extension via truncate() doesn't work as expected, but file shrinking and normal read/write operations work fine.
**Status**: Known issue, not blocking basic file operations. Memory-efficient SQL-based implementation is working correctly for all other operations.
**Next Steps**:
- Check how the Haskell implementation handles file extension/truncation with null bytes
- Compare behavior between Haskell and Zig implementations
## Testing & Verification Needs
**Test Suite Requirements**:
- Create comprehensive test suite that compares behavior between Haskell and Zig implementations
- Test all FUSE operations: read, write, create, delete, truncate, getattr, readdir
- Test edge cases: empty files, large files, binary data, special characters
- Test concurrent access patterns
- Test error conditions and errno codes
**Fuzzing Requirements**:
- Implement property-based fuzzer to generate random file operations
- Compare outputs between implementations for identical operation sequences
- Fuzz with random data including binary content, null bytes, unicode
- Fuzz file paths with special characters, long names, nested directories
- Fuzz operation sequences: create→write→read→truncate→delete chains
**Implementation Strategy**:
- Use same SQLite database file for both implementations
- Mount both filesystems and perform identical operations
- Compare file contents, directory listings, and metadata
- Capture and compare debug output and error codes
## FUSE Filesystem Testing Methodology
**Testing Approach**: Mount actual FUSE filesystems and interact with them through standard file operations, rather than testing individual functions in isolation.
**Benefits of Mount-Based Testing**:
- Tests the complete FUSE integration, not just individual callbacks
- Catches issues with FUSE protocol handling, caching, and state management
- Tests real-world usage patterns that applications would use
- Validates errno codes and error handling as seen by userspace
- Tests concurrent access and file handle management
**Testing Infrastructure Needs**:
- **Mount Management**: Automated mount/unmount with cleanup on test failure
- **Isolation**: Each test gets fresh mount points and database state
- **Parallel Testing**: Multiple mount points for comparing implementations
- **Process Management**: Handle backgrounded FUSE processes safely
- **Cleanup**: Reliable unmounting even when tests crash or hang
**Test Categories**:
1. **Basic Operations**: `open()`, `read()`, `write()`, `close()`, `unlink()`
2. **Directory Operations**: `opendir()`, `readdir()`, `mkdir()`, `rmdir()`
3. **Metadata Operations**: `stat()`, `chmod()`, `truncate()`, `utimes()`
4. **Advanced Patterns**: `mmap()`, `sendfile()`, `splice()`, atomic operations
5. **Error Conditions**: Invalid paths, permission errors, disk full simulation
6. **Stress Testing**: Large files, many small files, deep directory trees
**Testing Tools & Frameworks**:
- **Shell-based tests**: Use standard Unix utilities (`cp`, `dd`, `find`, etc.)
- **Language-specific**: Haskell property tests, Zig test framework
- **Specialized FUSE testing**: Tools like `fsx` (filesystem exerciser)
- **Performance testing**: `bonnie++`, `iozone` for filesystem benchmarks
- **Correctness testing**: `fstest` suite for POSIX compliance
## Future Enhancement: Column Type Flexibility
**Goal**: Support both BLOB and TEXT column types for `filename` and `filecontent`.
**Current State**:
- `filename` is TEXT (UTF-8 strings)
- `filecontent` is BLOB (byte sequences)
**Future Support**:
### Filename Column Types
- **TEXT filenames**: UTF-8 string handling (current implementation)
- **BLOB filenames**: Raw byte sequences for maximum flexibility with unusual filesystems
### File Content Column Types
- **BLOB columns**: Byte-based SUBSTR/LENGTH operations (current implementation)
- **TEXT columns**: Character-aware operations with proper UTF-8 handling
### Implementation Strategy
- **Detection**: Query `sqlite_schema` to determine actual column types
- **Adaptation**: Use appropriate SQL functions based on detected types
- **Testing**: Ensure both implementations work identically regardless of column types
### SQL Operation Differences
```sql
-- BLOB operations (current):
SELECT LENGTH(filecontent) FROM files; -- Returns byte count
SELECT SUBSTR(filecontent, ?, ?) FROM files; -- Byte-based substring
-- TEXT operations (future):
SELECT LENGTH(filecontent) FROM files; -- Returns character count
SELECT SUBSTR(filecontent, ?, ?) FROM files; -- Character-based substring
```
**Benefits**:
- Allows users to choose appropriate types for their data
- TEXT columns more readable in database tools
- BLOB columns handle arbitrary binary data more naturally
|