Skip to content

Fix BIT column in unique key bug#1483

Open
meiji163 wants to merge 9 commits intomasterfrom
meiji163/bit-key-bug
Open

Fix BIT column in unique key bug#1483
meiji163 wants to merge 9 commits intomasterfrom
meiji163/bit-key-bug

Conversation

@meiji163
Copy link
Copy Markdown
Contributor

@meiji163 meiji163 commented Dec 19, 2024

Related issue: #1480

BIT column values aren't compared correctly because gh-ost stores them as binary strings. Instead they should be compared as 64-bit unsigned integers (https://dev.mysql.com/doc/refman/8.4/en/bit-type.html). The fix is to convert them to uint64 in convertArg and make sure convertArg is called for all range queries.

In case this PR introduced Go code changes:

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

@meiji163 meiji163 added this to the v1.1.8 milestone Dec 20, 2024
@meiji163 meiji163 marked this pull request as ready for review April 30, 2026 22:49
Copilot AI review requested due to automatic review settings April 30, 2026 22:49
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes incorrect BIT unique-key range comparisons by normalizing BIT values to unsigned integers and ensuring prepared range queries compare them in numeric context, preventing infinite chunk loops / missing rows during rowcopy.

Changes:

  • Add BitColumnType detection (via information_schema.columns) and convert BIT values to uint64 in convertArg.
  • Render BIT prepared placeholders as cast(? as unsigned) in range/other prepared comparisons.
  • Normalize min/max/iteration range values after scanning so range queries consistently use converted args; add unit + local integration coverage.
Show a summary per file
File Description
localtests/bit-unique-key/create.sql Adds a local integration scenario with BIT in a composite PK.
go/sql/types_test.go Adds a unit test for BIT argument conversion to uint64.
go/sql/types.go Introduces BitColumnType, converts BIT args to uint64, and adds ColumnValues.NormalizeValues.
go/sql/builder.go Ensures prepared comparisons for BIT use numeric casting (cast(? as unsigned)).
go/logic/inspect.go Detects BIT columns and marks them as BitColumnType.
go/logic/applier.go Normalizes scanned min/max/iteration-range values so range queries use converted args.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 6/6 changed files
  • Comments generated: 3

Comment thread go/sql/types.go
Comment on lines +356 to +358
for i, col := range columns.Columns() {
this.abstractValues[i] = col.convertArg(this.abstractValues[i])
}
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NormalizeValues assumes len(this.abstractValues) >= columns.Len() and will panic with an index-out-of-range if called with a mismatched ColumnList (or a ColumnValues created with the wrong length). Consider adding a length check (and returning an error) or iterating up to min(len(values), columns.Len()) with a clear failure path when they differ, so this can't crash the process unexpectedly.

Suggested change
for i, col := range columns.Columns() {
this.abstractValues[i] = col.convertArg(this.abstractValues[i])
}
cols := columns.Columns()
limit := len(cols)
if len(this.abstractValues) < limit {
limit = len(this.abstractValues)
}
for i := 0; i < limit; i++ {
this.abstractValues[i] = cols[i].convertArg(this.abstractValues[i])
}

Copilot uses AI. Check for mistakes.
Comment thread go/logic/applier.go
Comment on lines 807 to +810
this.migrationContext.Log.Infof("Migration min values: [%s]", this.migrationContext.MigrationRangeMinValues)
if this.migrationContext.MigrationRangeMinValues != nil {
this.migrationContext.MigrationRangeMinValues.NormalizeValues(uniqueKey.Columns)
}
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BIT normalization is applied for min/max range reads, but the resume-from-checkpoint path populates MigrationIterationRangeMinValues/MaxValues from _ghk via ReadLastCheckpoint() and those values are never normalized. Since the checkpoint table uses the original column types (including BIT), scanned values will still be []uint8, which can reintroduce the incorrect range-comparison behavior (infinite loop / missing rows) after resume. Consider normalizing the checkpoint-derived IterationRangeMin/Max (e.g., inside ReadLastCheckpoint() before returning, or immediately after assigning them during resume).

Copilot uses AI. Check for mistakes.
Comment thread go/sql/builder.go
Comment on lines +61 to 63
} else if column.Type == BitColumnType {
token = "cast(? as unsigned)"
} else {
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new BitColumnType placeholder rendering (cast(? as unsigned)) in buildColumnsPreparedValues isn't covered by existing unit tests in builder_test.go. Adding a small test that builds a range/ checkpoint query with a BIT column and asserts the generated SQL contains the cast would help prevent regressions in this comparison fix.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants